Pere Ferrera Bertran

Datasalt

2011-05-07T02:54:00.000-07:00

I like to start new exciting things in spring. I think it's the best season of the year. It's not that cold to hide at home and it's not that hot yet to rest all day long.

After several years working at different Internet companies I've decided to start my own one. I am creating a company with Iván de Prado around the "Big Data" market which is called Datasalt.

Simply put, we're going to add some salt to your data. We'll analyze, transform and play with any amount of data thanks to the cloud and technologies like Hadoop, SOLR, Cassandra, MongoDB.

We'll be working on new products and tools that'll help us, our customers and the open-source community.

Checkout our spanish and english blogs!

Linkody

2011-05-07T02:49:00.000-07:00

How do you check whether your site is still being linked by site X? Is there a way to handle the status of several backlinks automatically? Yes, there is! Just check this "backlink checker" out. It's called Linkody. It'll handle it all for you and notify you when that bad friend just removed a link of yours from his site.

Percolator

2010-11-02T07:46:00.001-07:00

Big things happening. Google described how they implemented an ACID-Compliant distributed transaction system on top of BigTable. Some people out there claiming Map/Reduce is dead just because Google is not using batch processing anymore for indexing. The fact is that Percolator is not a general-purpose system, i.e. it can't do joins.

It kind of recovers the old idea of triggers - just that now triggers can also scale horizontally. Plus these triggers are written in a 21th century language, not in PL/SQL - BigTable and Percolator communicate through RPCs.

I don't think Map/Reduce is dead. It will still be an elegant programming model suitable for most data processes.

However speaking about real-time, huge data sets: things are changing.

Dream awareness and its complexity

2010-09-13T04:45:00.000-07:00

Probably you'll know about lucid dreaming. If not, it is a very interesting matter worth reading about it. Despite all the mythology and pseudo-science surrounding it (astral bodies, astral traveling and such), lucid dreaming is a scientific topic firstly investigated by PhD Stephen LaBerge.

There's been this recent movie called Inception that kind of has to do with it - but frankly I'm not strongly recommending it.

From time to time I happen to be 100% aware that I'm dreaming. More rarely, these dreams last long and are very detailed and real. I probably could count with my fingers the times this has happened to me.

Last night I had one of these. After realizing I was dreaming, I ended up exploring the world that my subconscious was continuously creating and doing random stuff to see how physics seem to be broken in dreams. I remember me throwing a pen and watching it 'fall up' instead of falling down. There was an old vintage TV that I wanted to break with an ashtray and the ashtray disintegrated itself when colliding with the TV - isn't it a beautiful metaphor for 'old things never die'?

Now there's the interesting part of the dream. I wanted to test the creativity of my subconscious by rapidly moving from room to room in the dream flat I was in. Every new room I discovered was more of an opened space and I was slowly moving somewhere outdoors. Every new space I discovered was larger and more detailed. I finally ended up in a huge field with bridges, walls, rivers, lakes, trees, mountains and hundreds of thousands of people walking, sitting, talking, smoking or having a drink. I slowly walked through this massive event and asked to myself: "Where do all these faces come from? Is my mind kind of creating them on-the-fly? Is my mind recovering faces from a database of random people who walk in the street everyday?". I was shocked. If I was asked to draw something beautiful in a paper, it would probably take me years to recreate all what I was contemplating. And it was all happening real-time.

The dream ended after I talked to a random blonde girl. She was normal except for the fact that her eyes were drawn manga-style instead of them being real. I told her: "You have weird eyes". And the dream ended.

Plume

2010-09-12T04:42:00.000-07:00

In my spare time I am collaborating to the Plume project:

"Plume is a (so far) serial, eager approximate clone of FlumeJava. The intent is to experiment with the design of the API both to understand the design decisions the Google team made and to see if there are good alternatives."

I first knew of FlumeJava thanks to Ivan. After I read Google's paper I thought "if this was implemented, it would be really usefull". I thought about beginning a prototype on my own, just to experiment. Just when I was about to write the first lines, I opened my reader and found this blog post from Ted Dunning. It was just like its blog title: 'Surprise and Coincidence'!

It's my first open-source collaboration and the closer we get to a first working version, the more exciting it gets.

Stay tuned, you hadoopers!

We are information

2010-07-19T05:03:00.000-07:00

Do you know which is the point in cryonics? It is to avoid the information-theoretic death of our brain. So, we are but information.

Then I thought: if the point is to keep our information and we are but information, couldn't we have another - more scalable - way of preserving it rather than having tons of frozen brains all over a room?

What if we did a dump of ourselves? What is the information we have to dump?

First of all, we need a dump of our DNA. Then, we need to dump the actual state of our brain to not loose our acquired knowledge and personality. We can nowadays express the DNA as a sequence of bytes (althought it seems there is some discerning on it). But what about our brain? Will computational neuroscience or some other discipline be able to transfer someday the state of our brain to a sequence of bytes?

Can you imagine the power of being able to save the information in your brain to a computer, modifying it and then being able to assimilate it again?

- You could apply patches. For example, you could apply the patch to modern times depression and instantly overcome it.
- You could backup important memories just as you backup your pictures. You wouldn't need to write down everything that happens in your life to a notebook anymore!

On the other hand:

- Your hidden thoughts won't be hidden anymore. You would have to encrypt them.
- Important organizations could steal brain data to extract information, predictions on aggregate people wishes, political views, and more. Exactly what they do now, just that it would be easier.

Current cryogenic institutes have a replication factor of one just because there is only one copy of the DNA and the brain state of a human being - the brain itself. Understanding ourselves as an array of bytes opens the door to the usage of modern IT methods for persisting human beings.

I wouldn't say it is impossible that we can restore a human being from a backup in the future just like we restore documents now.

"Damn, a terrorist had put a bomb in my car and I died yesterday. I had to restore myself from last week's backup"

Software engineering

2010-06-16T12:29:00.000-07:00

Should software development be engineering? We have enough knowledge nowadays to speak about software engineering. Practices used in design, code construction, integration and QA can be scientifically applied towards practical ends.

I have seen people using the word "engineering" as a synonym of bureaucracy, as if the concept of software engineering was opposed to the use of modern software development methodologies like RAD. Take this essay from Martin Flower with the quote "[...] reaction to the bureaucracy of the engineering methodologies [...]" as an example.

However, wikipedia comes to the rescue with a sentence that I consider valid: "[...] This distinction is misleading, as it implies that agile methods are 'unplanned' or 'undisciplined'. Further, agile teams may employ very highly disciplined formal methods [...]"

I believe brilliant projects start by proper specs, design and continue with a strong test-driven coding.

Unfortunately, nowadays most applied methodology is still "code and fix". Specially within strong market-driven companies that ask their developers to provide prototypes with very short deadlines - just for the sake of releasing something before their competitors.

Computers versus humans: feelings and emotions

2010-06-16T11:47:00.000-07:00

In Barcelona we have a metro line that is entirely driven by computers. I find it funny that in general we, as humans, tend to rely more on human drivers than computer-assisted ones. I think it is a cultural matter that arises from ignorance and that we will pull through over the next generations. It is probably due to human speciesism too.

Now, from my point of view there is one critical thing that differentiates us from computers: feelings and emotions. These may bring a human to poorly perform in any task. I see them as counter-productive routines written in our brain that are triggered more or less often depending on our nature as individuals. Can you imagine what would happen if computers had feelings and emotions influencing their activity?

- Imagine an emotional web service whom you send several incorrect requests in a row. It gets pissed off with you and decides not to return you anything in the next several requests...

- Imagine an emotional interpreter that has sentimental problems. It can't properly concentrate and will throw unexpected, non-deterministic errors to your scripts that you won't be able to debug...

- Imagine that two components of your server are angry with each other and decide not to talk to each other anymore for a while...

The magnitude of the influence of emotions and feelings in our work life is so big that good companies have to sort out all kind of problems related to them. They even might have personnel only dedicated to team building. Failure to create a good environment for positive human emotions to arise could lead to a company's disaster. What is even worst, is that our psychology tends to change very little with time and in the end it becomes a key component for recruiters to make the appropriate decision.

Diego Campo has a nice theory about human performance in sports where feelings pay an important role.

Virtual Theremin

2010-03-31T10:05:00.000-07:00

I have been working in a prototype of a synthesizer that converts the output of a webcam into sound. The name 'Virtual Theremin' is not very original as there appears to be other similar things exactly named like that - I am very bad for names. I will talk about them later.

You can see how the VT looks like in this demo videos I just recorded.

So how does it work?

To illustrate how the VT is implemented, let's imagine our friend Felix the cat is the input of the VT in a certain moment in time. Then, the state of the wave function will be obtained by computing two projections: the one in the X axis and the one in the Y axis. To compute the projections I just consider p(x) = sum(img(x, y)) for all possible y values, and viceversa. Each projection has different meaning for the computation of the wave form. The projection in Y is used to represent a spectral analysis whereas the projection in X is used to calculate a base frequency. For the spectral analysis I just discretize the projection to a reasonable limit (16 partials) and for the base frequency I compute the mean of the distribution of the projection.

Before doing the projections, I binarize the image and optionally apply gradient detection to it.

Hasn't someone done something like that?

I am aware that very similar things exist. There are very complex proprietary pieces of software that can do almost anything with live video. Speaking about simple software like mine, this and this are using a different approach: they are getting the input of the waveform inside the webcam output, whereas in the VT the whole webcam output is the input of the waveform. I think this is a big difference. It probably has been simpler to implement (otherwise you have to first recognize some kind of object to get some input) and it leads to infinite possibilities (any image can be the input of a waveform).

Finally, this thing implemented using the Wii is really cool.

If a painter was a computer engineer...

2010-03-17T15:15:00.000-07:00

Our friend painter in everything-as-computer-engineering-world wakes up every morning and starts painting at 9.30 AM. He knows what he has to do: he has to paint a certain picture. He has the picture in mind and he has been taught how to paint it. He starts painting and realizes that his favourite black pen is painting blue ink today. "Oh, I saw that happening sometime. It is not a reproducible problem, and I don't know what's going on. Last time I stored the pen in its box, opened it again and it painted black again". So our friend painter does what he thought it would work but, to his surprise, the pen is still painting blue afterwards. "Damn, there is something going wrong in here. I will check the documentation". The painter reads the documentation of his favourite black pen and searches for a section called F.A.P. (Frequently Arising Problems). He searches for the point "My pen is painting blue instead of black. What can be going on?". "Yeah, that's it, here we go". The documentation says: "We are aware that, under certain circumstances, this pen might paint in different colours than the one it is meant to paint. Please check for an updated refill".

Got it. Our friend goes to an ink shop and asks for an updated refill for his pen. He asks the seller, Mr Google, "What should I order? My pen is painting blue instead of black today". The seller opens a book by its first page and gives him several options:

Mr. Google - "Did you check the humidity of your painting room? Above 64% of humidity may cause your pen to convert black into blue ink. I have an updated refill that will avoid this problem at least above 40% of humidity.

Unsure of that, the painter goes back home and checks its humidity. 55%, nothing to worry about.

Mr. Google - "Did you dream with Russian women? Some people are experiencing blue ink after dreaming with Russian women. In this case you may want to wait for tomorrow, there is no known fix for that problem."

The painter hesitates for a while. "I don't remember what I dreamt!".

Mr. Google - "Do you feel rather depressed? Some depressed people have experienced random blue ink. To solve this issue, you have to draw using your left hand until the black colour is back."

The painter thinks about it. "I could destroy what I've done so far by trying to paint with my left hand. And I can't depend on drawing with my left hand: that's not a scalable solution!".

Mr. Google - "If you are painting inside a 32 square meters room, moving to a 64 square meters room may help".

"OK, but my flat is not that big...".

Mr. Google - "Does this pen really fit your painting project? There are too many opened issues for it. Maybe you should consider using a different pen?"

"Damn! I would have to repaint everything from scratch! In everything-as-computer-engineering-world, it is not possible to paint something with one pen and then continue painting it with another pen."

Mr. Google looks at the second page of its book - "Did you hear a man laughing downstairs while you painted blue for the first time? In that case there is a known fix for the problem."

"Ah, the man laughing downstairs. It remains me of something. What should I do?".

"Just go change the blue refill he put in your pen by a standard black one. And think about a revenge".

Virtual or real?

2010-03-02T11:17:00.000-08:00

Since almost a year, I have been buying some effect pedals for my instruments. I am slowly building a real daisy chain. Someone could think: what do you need that for? You can buy one Guitar Rig and you'll be able to create any possible effect chain, including effects and amp/microphone emulations that you'll never buy in your life! Well, this is kind of true. In fact, I have been using guitar rig for a while. The thing is, I got sick of manipulating virtual interfaces. I like to press real buttons, watch real leds turn on and use real smooth knobs. A virtual interface for that will never be as pleasant to use as the real one.

Conversely, there are many things whose experience improve in their virtual version. Even though I don't understand it, it appears virtual lectures are better than real ones.

And Mark peace in this article's ending is right, children of the future will experience virtual things pretty much the same as we experience real things now. So, probably my son will plug, unplug and tweak all sort of virtual effect pedals; to him they will appear as if they were "real".

Regarding the simulation hypothesis

2010-02-08T07:51:00.000-08:00

There appears to be a lot of bad arguments against the simulation hypothesis. As an example, once I read someone saying that simulated realities should be of lower complexity than the reality which is simulating them. I think it is not true, and that self-replication is only one example. There is also this Von Neumann universal constructor which not only can reproduce itself but it can also grow in complexity with time.

I went to wikipedia to read more about the simulation hypothesis and found a list of "However"'s that surprised me a lot. I will comment them all:

- "The assumption of this theorem is very far from being complete [...]"

Not very good for the first one: a negative-proof fallacy.

- "Arguments against simulation realism: Gravitational singularity, any of scientifical singularities [...]"

I don't see why an implementation of gravitation could not return Infinity for a certain combination of space and time coordinates.

- "[...] Weak helpfulness of Weather simulation, weak helpfulness of Earthquake prediction [...]"

I don't quite understand this one. The fact that current weather simulators could improve doesn't seem to say nothing against or in favour of the simulation hypothesis.

- "[...] Butterfly effect [...]"

Is this saying that simulations have to be linear? Aren't non-linear systems allowed?

- "Simulations are sometimes modified during their lifespan, but Physical constants (exact), Mathematical constants (exact), Exact trigonometric constants are stable (not to be confused with programming constants which are changeable)"

This one is a bit blurry. First of all, programming constants are not generally changed once the program is compiled. All programs have some exact constants declared whose value doesn't change unless a hacker modifies the exact byte of the constant in the compiled code. Secondly, this argument is a fallacy; the fact that some simulations are modified during their lifespan doesn't mean that there can't be simulations whose foundations don't change at all.

- "Simulations are temporal ("can be turned off", "restarted"), so the longer we (mankind and/or a person) live, the less likely it is that we are part of a simulation"

That is assuming that: "If a simulation is turned off or restarted, agents inside the simulation will notice it". I don't think that needs to be true. Maybe the universe just restarted now after being applied a patch and we didn't notice.

- "Simulation that is undetectable for scientists (for instance both astronomer and at the same time quantum physicist) would have to be very large and detailed, and hard to motivate with mankind ethic."

This is a misunderstood in complexity. Complexity doesn't only arise from very large and detailed systems, it can arise from very simple ones. Exact examples for this are fractals and, in general, any non-linear system.

- "Theory about simulation undetectable by scientists is not approved by ockham razor principle."

As if the Ockham Razor was a dictator who says what is true and what is not! Even though, it is the best argument in the list.

I think there isn't any reason why a system could not be simulated. Universally accepted Church-Turing hypothesis says Turing machines can execute any algorithm, and any digital computer is already a Turing machine given that it had unlimited resources. And in practice, none algorithm needs unlimited resources.

However, is the number of available computational resources going to become a singularity ever?

Algorithmic music generation

2010-01-27T07:20:00.001-08:00

Because I have studied music as well as computer engineering, I am interested in all topics that involve computer science and music. I have researched a little bit on algorithmic composition and created a tool that can generate music based on cyclic series. It is inspired by the theory of serialism, specially the theory behind dodecaphony.

The tool pretends to answer the question: can beautiful music be created by only using the rules of serialism?

In short, what this tool does is generate music by using a simple rule: repeating a series of notes in such a way that none of the notes is more important than the rest. You generate a series of notes and the program will play it according to some parameters. You can shuffle the serie from time to time, transpose it, use retrograde or inverted series and even assign basic series of durations and intensities.

The resulting MIDI file can be opened with any music editor and mixed with your favourite instruments.

When doing algorithmic composition programming, there is always the risk of ending up creating yet another random music generator (though there are some funny random music generators out there, and even serious projects with awesome mechanics involved). In my case I decided to create something simple and controllable, so that you always know what's going on and can therefore change parameters based on something, not just on pure chance.

What if ... ?

2010-01-13T12:27:00.000-08:00

One of the coolest thing the human being has is the ability of formulating questions and hypothesis. Sometimes, though, they are completely useless.

- Inconsequential questions: What if the universe was a simulation? Well, it is funny to imagine, but it is completely inconsequential: from our point of view life goes the same way with or without it being it a simulation.

- Hidden questions: The question 'What if a benevolent god created the universe?' is hiding a funnier one: 'What if an evil creator did the universe?'. If you think about it, if there was a creator, it is more likely to be a cruel god than a kind one (not a citation, but something I remember from Peter Cave - Can a robot be human?).

- Badly-defined questions: 'What if machines could think?', 'What if machines became intelligent?'. First of all, what does thinking/being intelligent mean? Even if we built a machine that mimicked a human, someone could always say that it is rather a machine, not a human.

- Questions that have no answer: 'Who am I?', 'What is life for?'. Evolution brought us the ability of questioning things, but we don't need to question everything! Even though, it is strange to think that some questions just don't have an answer.

Is the co-presence of man and the universe absurd? - (Funny how Camus defines religion as a philosophical suicide)

The human being: when a system becomes too complex

2009-12-16T12:07:00.000-08:00

It is said that human beings are wise enough to save the earth. The fact is that we are doing not more than destroying it. Our complexity can save the world, but first of all our complexity devastates our environment. Wouldn't it be easier if we were just a bit simpler?

Is the human being a 'system' which became too complex?

In my few years as engineer I have already witnessed the dead of systems because of their own complexity. Normally systems begin simple and small, they can be fully tested and easily monitored; they don't hurt anyone. But the process of incrementally adding components - and thus complexity - to the system is a very delicate one. When this is not done carefully, the system begins to be unstable, hard to mantain, impossible to modify and very difficult to debug.

When an engineer confronts a system which has became too complex, the smart solution is destroying it and replacing it by a simpler one. In other words, complexity is most of the times a disadvantage, not an advantage.

Sometimes I think about how many complex things we have developed and how much complexity we go through in our daily routine and wonder if it wouldn't be better to be a simpler specie.

What is my cat thinking when he stares at me while I am thinking about this all? Nothing. That's the power of simplicity.

Ending quote from Matrix:

"I'd like to share a revelation that I've had during my time here. It came to me when I tried to classify your species and I realized that you're not actually mammals. Every mammal on this planet instinctively develops a natural equilibrium with the surrounding environment but you humans do not. You move to an area and you multiply and multiply until every natural resource is consumed and the only way you can survive is to spread to another area. There is another organism on this planet that follows the same pattern. Do you know what it is? A virus. Human beings are a disease, a cancer of this planet. You're a plague and we are the cure."

P.D. - Funny how human complexity can bypass natural selection according to this article. Not even the wisdom of our environment can destroy us.

Life as a search algorithm

2009-11-20T09:31:00.000-08:00

Uninformed search

It is the simplest method and probably the less used one. I often find people building complex heuristics around things that can be simply solved by brute force. Why would you try to guess which one of these three girls will like to have a date with you? The search space is small enough, just try asking the three of them!

Simulated annealing

Example #1: A situation where you drink too much. You will try to optimize the fun you get from the people in the party. At the beginning of the search, the 'temperature' of your drunkness is high and you will tend to switch conversations from one person to another, even if the second person is not as interesting as the one you were talking with. When the night passes by, your drunkness gets colder, you get depressed and you tend to just stay with the person you have nearby.

Example #2: Relationships. Isn't it easier to quit a relationship in the very beginning of it rather than quitting after three years?

Greedy algorithms

Why do people try to pack things into a bag using greedy strategies? Things like picking the bigger item at each step and putting it in the bigger hole available. It just doesn't work! It's the kind of NP problems that don't accept greedy strategies!

What about life? In life we try to maximize the accumulated pleasure throughout the lived years. We usually take a greedy approach for solving the pleasure of life, but a lot of people find real happiness by introducing random deviations in the search space, for instance, relocating in a different country for no apparent reason.

About simplicity

2009-11-13T01:31:00.000-08:00

The older I get, the more I tend to value simplicity. Engineers like me are already taught in early stages of their learning processes that good solutions are often simple ones. Nonetheless we keep increasing the complexity of problems which are, by nature, far simpler.

Not that I can consider myself too old, but situations where I recall past mistakes arise more and more often. Thoughts like 'No, this ain't going to work' or 'I have to think about a simpler approach', 'This whole thing is getting too complex'. Not only in engineering aspects of my life, but also in musical and social ones.

Do young people make things too complex because of their natural amount of energy? Is this energy lost with time - mainly because of failed experiences - or does it simply flow in a different, wiser and more controlled way?

I remember in my first years of learning piano the sensation of being frustrated just because 'I couldn't play that fast'. I wanted to play all sorts of complicated rhythms, harmonies and melodies. Today I would just be happy of being able to record a warm sound - one that is warmer enough.

True complexity seems to be hidden behind neat simplicity.

P.D. - You can check the "Simplicity proof-of-concept" experiment I did in a programming contest. I created a super-simple football team which happened to be better than 75% of the teams in the contest.