The Filter Bubble: what the internet is hiding from you…

So our web experience is somewhat customized by our browsing history, social graph and other factors. Should we care that this sort of information-tailoring takes place. Eli Pariser, founder of public policy advocacy group MoveOn.org, explores the topic in The Filter Bubble.

Personally I have a load of issues with the whole concept that the internet is worse than what we already have. We all have filters, as irrespective of the amount of information; we filter, pick and choose things we like, align to self interest, motivate, warm to, find interesting, our background, our traumas and use friends, family, beliefs, faith to determine what we think and listen to. Least we forget the moment in time when this occurs, other distractions, stresses and items competing for our attention.

The concept that the internet filters based on what we view as a filter is better or worse than picking a TV channel, news paper or magazine – which itself has a editors who picks what they like and we pick what we like, it’s a filter.   If we did not pick what we like – we would never have facts to back our opinion. We like confirmation of what we like – the issue is not about the filter but is about understanding who we are and why we think like we do.  This has nothing to do with code!

As per another post on MyDigitalFootprint – I want to know your opinions and not the facts as I can always find a fact to back my opinions – your opinions are your filter.  Google (and all other search engines) also don’t allow you to overly filter and does make serendipity possible. Follow the writings at http://searchengineland.com/ if you want more about this.

What does worry me is that the coder may bias my views based on their interpretation of the algorithm – but this has nothing to do with the Internet.

There is a great article here about Gerrit Visser who has been digitally curating content since “just after the internet was invented” in 1996.

Quoting

“I think the curator (not the strategist) will have four main roles:

1. Searching, filtering and selecting content to become a taste-maker for the target audience.
2. Providing curatorial leadership to help other workers within an organization understand what makes valuable content for the brand — so they can be enlisted to create and maintain content based on these evolving criteria.
3. Spotting trends, and feeding these to the strategists who will use them to help define future direction.
4. Distributing — identifying channels and fine-tuning them.”

Looks a lot like what I try to achieve here at blog/My Digital Footprint and I massively depend on recommendations and comments from others …..

Here is the thinking, which has wider implications than a small change at Google….

If you took a complex algorithm and asked a 15 year old, a 30 year old and a 65 year old; both male and female, from different countries, using different computing languages and compliers to cut some code: will you get the same output from the same test datasets using the different implementations of the algorithm? – Probably not!

So changing the algorithm is one thing; changing compliers (and who coded that), language and the age, sex, experience (life and skills) of the coders is another……but we depend on them.

Yes there are tools to help ensure maintainability, supportability, scalability, performance and conformity but we do have a massive and increasing reliance on the coders ethics and lack of bias in the way the interrupt an algorithm……just wondering who is thinking about this as well.

Why this is important to digital footprints. Someone you don’t know is taking your data and predicting your future based on their and others bias…..

-------

From the Google blog Ten recent algorithm changes

11/14/11 | 8:30:00 AM

Today we’re continuing our long-standing series of blog posts to share the methodology and process behind our search ranking, evaluation and algorithmic changes. This summer we published a video that gives a glimpse into our overall process, and today we want to give you a flavor of specific algorithm changes by publishing a highlight list of many of the improvements we’ve made over the past couple weeks.

We’ve published hundreds of blog posts about search over the years on this blog, our Official Google Blog, and even on my personal blog. But we’re always looking for ways to give you even deeper insight into the over 500 changes we make to search in a given year. In that spirit, here’s a list of ten improvements from the past couple weeks:

• Cross-language information retrieval updates: For queries in languages where limited web content is available (Afrikaans, Malay, Slovak, Swahili, Hindi, Norwegian, Serbian, Catalan, Maltese, Macedonian, Albanian, Slovenian, Welsh, Icelandic), we will now translate relevant English web pages and display the translated titles directly below the English titles in the search results. This feature was available previously in Korean, but only at the bottom of the page. Clicking on the translated titles will take you to pages translated from English into the query language.
• Snippets with more page content and less header/menu content: This change helps us choose more relevant text to use in snippets. As we improve our understanding of web page structure, we are now more likely to pick text from the actual page content, and less likely to use text that is part of a header or menu.
• Better page titles in search results by de-duplicating boilerplate anchors: We look at a number of signals when generating a page’s title. One signal is the anchor text in links pointing to the page. We found that boilerplate links with duplicated anchor text are not as relevant, so we are putting less emphasis on these. The result is more relevant titles that are specific to the page’s content.
• Length-based autocomplete predictions in Russian: This improvement reduces the number of long, sometimes arbitrary query predictions in Russian. We will not make predictions that are very long in comparison either to the partial query or to the other predictions for that partial query. This is already our practice in English.
• Extending application rich snippets: We recently announced rich snippets for applications. This enables people who are searching for software applications to see details, like cost and user reviews, within their search results. This change extends the coverage of application rich snippets, so they will be available more often.
• Retiring a signal in Image search: As the web evolves, we often revisit signals that we launched in the past that no longer appear to have a significant impact. In this case, we decided to retire a signal in Image Search related to images that had references from multiple documents on the web.
• Fresher, more recent results: As we announced just over a week ago, we’ve made a significant improvement to how we rank fresh content. This change impacts roughly 35 percent of total searches (around 6-10% of search results to a noticeable degree) and better determines the appropriate level of freshness for a given query.
• Refining official page detection: We try hard to give our users the most relevant and authoritative results. With this change, we adjusted how we attempt to determine which pages are official. This will tend to rank official websites even higher in our ranking.
• Improvements to date-restricted queries: We changed how we handle result freshness for queries where a user has chosen a specific date range. This helps ensure that users get the results that are most relevant for the date range that they specify.
• Prediction fix for IME queries: This change improves how Autocomplete handles IME queries (queries which contain non-Latin characters). Autocomplete was previously storing the intermediate keystrokes needed to type each character, which would sometimes result in gibberish predictions for Hebrew, Russian and Arabic.

If you’re a site owner, before you go wild tuning your anchor text or thinking about your web presence for Icelandic users, please remember that this is only a sampling of the hundreds of changes we make to our search algorithms in a given year, and even these changes may not work precisely as you’d imagine. We’ve decided to publish these descriptions in part because these specific changes are less susceptible to gaming.

For those of us working in search every day, we think this stuff is incredibly exciting -- but then again, we’re big search geeks. Let us know what you think and we’ll consider publishing more posts like this in the future.

Posted by Matt Cutts, Distinguished Engineer

Background

Trust and Brands are interwoven like the double helix of DNA. A Brand is much more than the image, logo, name, awareness, experience, campaign, product or trademark. Whilst all of the above (and more) are essential components of a brand, the brand itself is the meeting of an ‘intention’ and a ‘promise’, a confluence that involves Trust. A recent Interbrand survey valued Coca-Cola at US$73 billion, Microsoft at US$70 billion and IBM at US$53 billion. Underpinning that value lies the experience the brand provides to it’s customers. The consumer experience comprises many things.

Today, the iPhone is a textbook case of a brand leveraging a consistent customer experience for it’s customers. The iPhone, and many other leading brands, provide both the experience and the Identity for the customer. My favourite example of a brand with timeless trust is Patek Philippe. The watch is called a ‘chronograph’. There are no prices on the site as far as I could see. The advertising shows that the watch is yours to experience and to ‘hand down to your child’ i.e. a tradition/legacy. This type of branding and the promise it depicts is truly timeless and will remain so.

Thus, Trust is beneficial for brands. Brands want to be trusted and indeed, some are trusted reflecting their market value.

But at the same time, the values, traditions and norms of society are changing and brands are reacting to that change. We see this in many ways – for example – Brands are displaying their “environmentally aware” credentials in response to greater awareness among consumers. The 2011 Edelman survey on Trust ranks financial institutions at the bottom of the ‘Trust’ scale. The survey  also indicates  attributes like Quality, transparency, Trust and Employee welfare as valued attributes by customers. It goes even further by finding that reputation enhances believability i.e. customers have to hear something about a specific company multiple times for them to believe that information. 26% have to hear the message 4 to 5 times and 59% have to hear it 3 to 5 times. In an era of current media scepticism, customers are influenced by multiple voices and multiple choices and the need for authority and accountability set new expectations for Brands.

The Algorithm lens, the Local lens and the changing balance of Power

However we define Trust, we acknowledge that Trust is  a two way processes.  Brands need Trust  and indeed customers trust some brands which is reflected the high market value of the best known brands. However, the nature of Trust in a brand is changing. The Web has led the first phase of this change as customers have become active and vocal. They are no longer passive consumers. The information they contribute transforms their relationship with brands and in some cases the Brand itself. Beginning in 2005 with the emergance of the Web 2.0 generation, two shifts happened: Firstly, Customers contributed data .Secondly, search engines harnessed that very data to create a ‘filter’ for our online world based on our data. Today, we are living the Social or Facebook era which takes the sharing of data to the personal level and by extension, extends the filter to the our social graph

Increasingly, with the greater availability of data, firms are simply ‘harnessing’ all this data which customers share. Today, the balance of power rests with the providers and with the firms which have the ability to capture data. We do not see the current generation (which we call ‘Screenagers’ – i.e. people who grow up interacting with multiple screens daily), sharing less data. On the contrary, the trend to share more will continue. We also see that companies will continue to harness that data and will provide more  services based on Data. This gives the perception that the balance of power has shifted away from the customers and towards the providers (such as Apple, Google, Amazon).

But the balance of power shift may not be so one sided.

Web orientated search engines put an ‘algorithm lens’ over online content. Mobility adds  a ‘local lens’ over both physical objects and online content. In other words, the web and mobile based search engines created rankings and thereby a filter or ‘lens’ for search results based on analysis outside of the control of the user. In a multiscreen world that the Screenagers inhabit, these multiple screens will be generative i.e. they will create their own data and by implication contribute to the  filters. This filtered data will be used by everyone, which means it could also be used by customers themselves. Customers will be able to see their world through this ‘lens’ of someone else’s data. Customers’ data could be harnessed by others but customers could also easily choose to share key data components and / or create a set of preferences that would ‘colour’ their world through their own data lens to their own benefit

This paradigm could bring back control to customers even when their freely available data can be harnessed by others. Thus, in this phase, all brands will be affected. Just like the Web 2.0 phase produced brands like Amazon and eBay, we will see the rise of new brands which will serve the customer especially when the Screenager mindset will be the dominant paradigm for societies (especially in cities) which affects our physical space. The foundation which drives this relationship will be data.

The Screenagers hypothesis

Data is already changing many areas that we once took for granted – ex Data driven journalism . Data is being released by governments, individuals and companies at a phenomenal rate. Over time, we expect that Open source, Open standard, Peer to Peer initiatives will arise to create this ‘lens’

So, the Screenagers hypothesis is:

a)    Brands will be expected to fulfil their promise but that will be only the minimal requirement

b)    Customers will continue to share data about themselves and about their brand preferences. This data will continue to be harnessed by providers. The rate of this behaviour (both sharing and harnessing) will increase

c)    Simultaneously, it will be possible for customers to harness their own data

d)    This data will act as a lens/filter for services

e)    This will profoundly change the relationship with brands and new brands will emerge to take advantage of this paradigm and serve the customer better

The screenagers event will explore this hypothesis in detail and will look at three axes i.e. Data, Benefits and Services to create a model to explore this hypothesis. We will explore these ideas in the Digital Footprint Summit Learnings and Insights from the Screenagers – Thursday, November 3, 2011 from 9:00 AM to 6:00 PM (GMT)

This TED talk  by Julian Treasure  - 5 ways to listen better. He has several versions on the same topic and theme http://www.juliantreasure.com/Julian_Treasure/Home.html  The reason for putting it here on a blog about digital footprints is that listening is part of our digital footprint...and sensors are trying to do what Julian describes for us to help in that filter.

This is driven by a thought - is there a flaw in personalisation?

Most TV service providers now recognise that there is a need to incorporate recommendation as part of their content discovery mix - according to TV Genius. Depending on the provider, this can be a matter of personalising the video-on-demand store, promoting premium TV channels, or driving viewers to video-on-demand services from within the traditional EPG. All of these solutions have appeal to different demographic groups, but recent research they have conducted shows that a much broader content discovery solution is required.  After all, not every user has the same exact content discovery needs; while some viewers know exactly what they want, others are simply browsing for something new to watch.

 So segmentation could look like ..... with each group having very different content discovery behaviours.

1. Socialites: Influenced by friends and family, channel surfing, and web and mobile

2. Progressives: Influenced by web and mobile content

3. Reactives: Influenced by channel surfing and on-air trailers

4. Traditionals: Influenced by newspapers, magazines, and on-air trailers

5.  TV addicts: Influenced by on-air trailers, the EPG, and web and mobile content  

So the question is does personalisation work or is it broken.....

In the book I looked at this problem and concluded that you needed two types of input/ feedback for personalisation/ recommendation to work.  Your own personal preferences (for improvement and refinement) and trends from a wider community (for colour, flavour and breath)

This is so much more eloquently put by Eli Pariser: Beware online "filter bubbles"   in his TED talk and read the comments at the end.

 However, I agree with the outcome - if we depend on the algorithm to much we are doomed.

"do as I say don't do as I do" from the Genesis Song "Jesus he knows me"

We tend to have different accounts. Facebook, MySpace, LinkedIn, Plaxo, Filckr, YouTube, Gropon, Blogger, Disqus and Twitter to name a few and we cannot forget Gmail, Hotmail, and a splatter of Skype and IM.  Each one created at a unique point in time, with a profile and either to test or to engage.

Each service and hence each account has a different audience and with each, you *probably* presented a different persona or in some cases names.  Personally I use different accounts to filter spam and junk and to see who sells my data.

I am trying to bring all my lists together and consolidate one list of contacts (currently 11,739) in the hope that in my brave new world I will have one list no longer in silos:  Facebook, Twitter, email contacts but will have one contact and all of their content, views, contacts as one digital person. But like me others have persona and details that they expose depending on the application. 

So please to make my life easy can you do what I will not do.  I want my life to be easy - therefore I create persona.  However, for me; if you do this - it makes my life complex.  Dilemma of 2011!

I wrote that Social filtering is deeply human at the beginning of November and I knew that there was more to the topic/ theme/ thought then but I could not articulate it. Since then I have been juggling with various ideas, these have often been driven by my necessity to justify Twitter. Twitter, get it or not, provides a function called “follow” – you can follow who you like, and you get updates/ insight/ information/ attention from them. However, can you turn “follow” into value and is following your social filter based on those you trust.

Follow has an obvious value to the person who follows the leader. You gain free insights/ selection/ value/ updates/. This social filter is based on trust and it is different from curators and editors who have specific agenda’s and income/ profit requirements. In the original post I quoted David Armano “Often times the quality of links and information I get on Twitter is better than what I would have gotten from Google because the knowledge of the human feed is deep, niche, and fickle.”

Scenarios

Here are several scenarios to consider when thinking how we could turn follow into value and comparing outcomes from search and social networking, they are not exhaustive but should provide a good place to start a train of thought.

1. I am looking for a great Thai restaurant

Option 1. Search.   Type in “Great Thai restaurant” into Google, my mobile sends my location and Google takes a guess I want food tonight and near to where I am search, reasonable assumptions driven from our need for context and personalisation. From the “unknown algorithm based results” that favours Google, I then read some third party reviews which I cannot judge if they are paid, biased or just vocal. Is the selection any better than walking past and seeing how many people are sitting in the restaurant?

Option 2. Post to Facebook and ask my friends and my network where a “Great Thai restaurant is” – there is more work to this one and I am wholly dependent on someone helping. Size of network helps at this point.

Option 3. Twitter/ follow. I love Thai and I am already following others who love Thai. I Tweet to my network of same minded followers who can deliver a recommendation. 

In option 1 – Google wins. In option 2 – Facebook wins. In option 3 – the community wins and the person who helped me may get a discount on their next meal.

2. I want to invest some money

Option 1. Search.   Type in “Great Investment fund” into Google. From the “unknown algorithm based results” that favours Google I will click on some links and read, subject to many legal notices, about the performance of various funds. If I invest I will have watch and wait for the results

Option 2. Post to Facebook and ask my friends and my network about their experiences with “Investment funds.” Not sure I would really be that happy with this for many reasons including telling the world about my desire to invest.

Option 3. Twitter/follow.  I love to invest and I am already following others who love investment. I follow a service that allows me to manage my own money (never give up control) and I invest based on what the best in class is doing (www.covestor.com) To follow the best investor I share some of the upside. No management fees, no overheads, risk on my terms, stop and start when I like. Worth noting that J.P Morgan funds investment advice is now on iTunes

In option 1 – Google wins. In option 2 – no-one wins. In option 3 – the person who I follow gets a share of my upside, assuming that they want to create value over time and not destroy it once.

3. What is hot in tech/ service/ my industry

Option 1. Search.   Type in “what is hot in tech” into Google. From the “unknown algorithm based results” that favours Google I will click on some links and read. The top tech web sites are there with breaking news. I can use various tools to determine what is hot and trending or I can use my “reader” to filter from my own favourites.

Option 2. Post to Facebook and ask my friends and my network about what they think is hot. Day 1; I will get a few views. Day 10; I will get a less help and probably a polite note telling me not to ask again. 

Option 3. Twitter/ follow. I look at what is trending and select a few “trusted” people to follow and follow updates as and when they occur. I add value to my network by adding my own opinion, or pay to sit there and listen.

In option 1 – Google wins. In option 2 – no-one wins. In option 3 – the community/ cluster wins.

Logical response

The obvious contention to these three and very simple scenarios is; to Quote Paul Rodriguez who commented,  “lemmings, pied piper, following somebody the wrong way up a one way street, jump off a cliff if I told you, following the falling domino in front and having the falling domino behind follow you, following somebody you trust, who is following somebody they trust who is following somebody they trust who is following somebody stupid, the list is endless...the risk is that instead of having the madness of crowds, maybe the 21st century equivalent is the madness of tweets? Laws such as the snowball effect and the law of unintended consequences become far more amplified in an interconnected world. In which case market (and wealth) fluctuations become more volatile, but then you only *truly* make money on the gradient.”

I expect that there is a lot of empathy for the logic of this response, however, is follow (Twitter or other tech based follow services) any different from what we have today with editors/ press/ celebrity and broadcast as we all believe everything from the red top tabloids and sky/fox news!

Context

However, putting follow into context Researchers at HP Labs discovered that Twitter can predict, with astonishing accuracy, how well a movie will sell. The researches at HP started by monitoring movie mentions in 2.9 million tweets from 1.2 million users over three months. These included 24 movies in all, ranging from Avatar to Twilight: New Moon.  Then they took two different approaches, dealing with two very different performance metrics: the first weekend performance, which is largely built on buzz and the second weekend performance, which is largely built whether people actually like the movie. To predict first weekend performance, they built a computer model, which factored in two variables: the rate of tweets around the release date and the number of theatres its released in. Lo and behold, that model was 97.3% accurate in predicting opening weekend box office. By contrast, the Hollywood Stock Exchange, which has been the gold standard for opening box-office predictions, had a 96.5% accuracy. “

What should be even more alluring to business strategists and CEO’s; as Tech Review points out, Twitter might be more than just a mirror of mass sentiment - the service might also influence it. In other words, could you actually make a product launch far more successful with a really smart Twitter/ Follow strategy?   However are we measuring or observing the results of a system in motion and in the process influencing those results? For anyone with a science background this will bring up Werner Heisenberg and The Uncertainty Principle

Heisenberg determined that “both the position and momentum of a particle cannot be known simultaneously.”   The dichotomy raises the mind-boggling prospect that unless we observe an event or thing, it hasn’t really happened, that all possible futures are quantum probability functions waiting for someone to notice them - trees falling unheard in a forest. Maybe this Viewpoint never existed until you searched for it and Google created it as you wanted it!

(Yes for those who have mastered QM I am confusing the observer effect of with the uncertainty principle. Technically the uncertainty principle has nothing to do with "observing", it has to do with measuring. The observer effect is a supposed effect of observing an event and the influence of your observations on the event. No one would ever have to actually observe a particle's position to obfuscate its momentum, the mere act of using the photons to measure its position, even if nobody ever observed it, would suffice. It's the act of measuring, not actually observing that causes the uncertainty principle, but when observation requires something that may cause change the problems occur)

Anyway, how does this relate to the analysis and feedback within my framework of thinking about Follow?  Think about it this way:  The mere act of observing a social change, changes the behaviour of that social object.  In “reality TV” they put cameras in front of “real” people for the viewer to watch how “real” people behave, date, compete, etc.  But this in fact makes those on camera less and less real.   They’re not actors, nor are they behaving like normal people.  They are somewhere in between the two. 

In the case of Twitter predicting a movie success, could an editor or critic have the same effect, if they could do it in real time and not on paper? How does Google real time search affect your searching habits and techniques.  You no longer have freedom in the web, as the recommendation is based on what the crowd says is important and therefore we are actually just lemmings.

Restating the Problem

Therefore the problem (Generating wealth from the web) is far more complex, multifaceted and inter-twangled, as there is unlikely to be a single source.

• Do I want to be directed by people I trust but I may not be able to determine their source – Follow
• Do I want to be directed by an unknown algorithm that can change at any time and could be biased to their own needs – Search
• Do I want to be directed by Brands – Marketing/Ads
• Do I want to be directed by the media/ editors/ critics where I may be able to determine their bias – Broadcast/ News
• Do I want to be directed by the fashion/ celebrity – Sales

This complex dependency is an issue which editors and bloggers have faced time over. Do I write based on what people want to read, based on clicks and response data or what I find interesting – are we (am I) adaptive or reactive, do we want to be individual or loved or make money or provide democracy or lead?

I really don’t need to know what you had for lunch and I don’t have to follow you.  Follow would put me in control and can seek out value from the community and not some bland algorithm that controls what part of the web I can see. However the issue facing follow is how will I pay the platform that underpins the service?

Effort

A reasonable concern would be that the 'follow' theory is weakened if the 'followed' account generates little content, or at the wrong time. e.g. If I follow five Thai restaurants but only one puts messages out, at 1am, I am not going to that delighted with the experience (unlike search that does not provide real-time as everything has to have been indexed). This low level of activity from Follow has two effects, you give up early or I personally have to exert a lot of effort as I need to continually add/prune/curate.  This takes time, as humans we are inherently lazy and would therefore prefer for someone else to do this for us.  Editors rule!

Wrapping up

This long Viewpoint started with the idea that “follow” is the new economic model poised to take on “search” and I believe that there is substantial value in “follow.” Reading that Google offered $3bn for Twitter makes be believe that there are other strategists who are struggling with the same issues and the value!

Social Filtering is Human - not brands, not curators, not search, not editors

In my book I write that there is limited value in the collection of data (it will be a commodity game), I contend that storage of data is just a cost and a liability, value will be derived from analysis and being about to control the feedback loop.  Analysis, I proposed, was algorithms – deeply complex bits of code that could draw out meaningful views from all the data.

15 months on from writing I am left thinking that one aspect of the algorithm – (social) filtering is actually a deeply human task as I trust my network more than brands, curators, search and editors.

Therefore, whilst I accept that algorithms can do the task and as well – will I ever trust the results?

To quote from David’s very good article, where the diagram came from as well

“Often times the quality of links and information I get on Twitter is better than what I would have gotten from Google because the knowledge of the human feed is deep, niche, and fickle.”