The blogosphere has been abuzz recently, or, at least, it seems that way if you've only been checking up on it sporadically the last few weeks. Jennifer Rohn's post about lab notebooks has spurred over 100 lively comments spanning electronic lab notebooks, peer-review, openness in science, and the reward system in science, making for an engrossing peek at the social science of science. Cameron's own musings on that discussion. Pawel Szczesny writes about what it means to be a freelancing scientist. All of this is fascinating and it is exciting to contemplate both what the future of science holds and the obstacles we will need to overcome; the fact that there are indeed stubborn obstacles (technological as well as cultural) and potentially tremendous rewards makes the anticipation of that future all the more heightened.
Emboldened by the collective fervor, I would like to propose an idea - an idea with the same name as this blog. But first, the back story.
About 8 months ago, one of my lab mates was writing up a short paper for submission to a translational bioinformatics conference. The work she was submitting revolved around a powerful literature-search tool tailored for pharmacogenomics called Pharmspresso. Although Pharmspresso had features lacking in existing search methods and was thus useful, the intent was for it to recognize genes, drugs and polymorphisms in free text, and so she needed a way to evaluate its performance. The evaluation task would be straightforward: given a set of pharmacogenomics papers, what percentage of the mentions of genes, drugs, and polymorphisms does Pharmspresso capture? Getting the list of recognized entities from Pharmspresso would be easy, just give it the documents and set it running. But what would be the gold standard?
Typically, gold standards are created by humans. In this case, it would be the list of entities recognized by human readers with the appropriate knowledge to make the distinctions, in the same set of papers. To get her gold standard then, she essentially asked favors of her colleagues in the lab and the department, which translated to a number of them reading papers and doing data entry during free time (or during faculty talks) at a departmental retreat in early fall - not exactly fun, but done out of a sense of duty to science and the goodness of their hearts.
Afterwards, while socializing during one of the poster sessions, this task came up, and the discussion (in which Samuel Flores, Magda Jonikas, Yael Garten, Alain Laederach, and Bernie Daigle all participated) quickly turned to alternative solutions for tackling this and similar problems in science - those requiring knowledge and resources external to your own. As another example, many bioinformaticians work on problems that produce predictions of functions which would benefit from experimental tests of their validity. Conversely, a wet lab may benefit greatly from someone with computational expertise guiding or leading the data analysis, or even providing the hypotheses for experimental studies (in the form of predictions). This is the stuff from which many collaborations are born, but it may be difficult to find the right people in the first place, or the task at hand might seem not quite collaboration-worthy.
In essence, the problem boils down to this: you or your lab possesses a certain collection of skills, knowledge, and resources (hereafter referred to as simply resources), but your needs may not be fully addressed by what you possess. The solution lies in this simple proposition: some other person or lab has what you're looking for.
While it makes sense for a lab or individual to grow their resources and be mostly self-sufficient, at some point it becomes more economical to outsource certain tasks - to companies for antibody development, software for data analysis, supercomputers for high-throughput computing, etc. In some cases, the exchange takes place directly at the academic level, for example, with some labs maintaining and sharing specific cell lines or mouse strains for use by other researchers, or less directly through the use of published and available tools for all sorts of tasks in bioinformatics. So it would seem that outsourcing is common and accepted. But aside from these sorts of established avenues, what other needs do scientists have in conducting their research that are not easily solved? How often is a line of inquiry abandoned or slowed because of a lack of necessary skills, knowledge, or material resources?
The idea behind One Big Lab is that the scientific community should act as, well, one big lab, sharing resources when it makes sense, and everyone, especially the community as a whole, benefits.
During that discussion at the departmental retreat, the solution boiled down to some form of online transaction service built around a credit system. Scientist X would like 5 gold standard outputs for a certain task, so she posts a description of the task along with some credit attached. Other users can then sign up to complete the task, after which they receive the stated number of credits. Of course, in order to post tasks, you need to have a balance of credits you can draw from - which you earn by doing other people's tasks. Getting credits into the system to start needs to be figured out (give everyone N credits? Money for credits?), but assuming there's some baseline of credit floating around amongst the various users, an equilibrium should eventually be reached (at least, that's the hope).
Variations on this theme are natural - have a peer rating system, have the final credit payment be subject to a bidding system (based somehow on user ratings, e.g. highly rated users can ask for more credits to complete a task and the task-poster may select which user to "hire" based on the user ratings as well as how much each user is asking), have some kind of mechanism for taking transactions "offline" into serious collaborations, etc. Tasks may run the gamut from routine and rote to intellectually stimulating and scientifically rewarding. Obviously, guidelines will have to be set for what transactions may be appropriate for this forum and which ones might be more suited for formal, collaborative relationships - but even here, a forum such as this could be very useful for finding collaborators.
In addition to the scientific transaction system, there could be other features that build on the community aspect, such as journal clubs, informal manuscript review, resources for students, and discussion forums. There could be repositories for knowledge or links to existing ones, informal or formal consulting, and casual exchange of ideas which could stimulate research or professional development. All of this should reinforce the idea that science is strengthened by community and the scientific community should not be held back by insufficient allocation of resources.
Although there are a number of websites out there that tackle some of these aspects, especially the community-building ones, I haven't really seen much resembling the transaction system, which is really the core of the idea. Pawel's freelance science comes close, and what I'd like to see is a formalized community-wide online service for essentially that. Maybe this is technically infeasible right now right the way grants work (it may be difficult to justify spending time or resources on other people's research) or with the way scientists work, but I would like to think that the basic premise - bringing together people with complementary skills and resources - makes sense and balances out in everyone's favor. (Whether this premise actually pans out in practice is up for debate - if we offered credits for cash, would anyone ever do someone else's tasks, or would demand outpace supply? By the same token, there could be "freelance" scientists like Pawel who primarily complete tasks, and could then have the option of "cashing out".) I'm sure there are a ton of tricky legal, IP, financial, organizational, etc not to mention social and cultural issues (would you trust someone you don't know to do work for you?), but I think the idea of having One Big Lab is worth exploring.
If I had the time, skills, and business acumen I would throw together a prototype and work out a business plan, but at the moment the most I can do is outsource it to the closest thing we have to One Big Lab - the blogosphere. ;)
Incidentally, Alain Laederach had come up with a similar idea about a year earlier and we thought about naming it "Experitrade" - an online system for trading experiments, essentially, but the name sounded too corporate and the grant he wrote never got off the ground. But the idea has persisted and inspired One Big Lab.
So, I'd welcome any thoughts, logical extensions, deal-makers or deal-breakers, important issues to consider, "prior art"... does anyone think this idea has legs? Will it work if it is completely altruistic? Does adding money into the equation detract from its mission or the science? What sorts of technical and organizational roadblocks are there? Clearly it makes the most sense, if any prototype is developed, to start small - with a couple participating labs or within a school or university, which helps with the trust issue as well. But I'd like to make sure I'm not completely missing the picture!
Subscribe to:
Post Comments (Atom)
19 comments:
I think there are real problems with the notion of 'paying' people with some sort of currency. To me it is better to focus on the mutual benefits, getting authorship, sparking new collaborations, and perhaps where there aren't tangible mutual benefits just actually paying cash for the effort.
That said, something like this is exactly what we need. Will try to post something on this and gather ideas together. Whether or not you have a currency exchange element the key is critical mass. The best way to achieve critical mass will be to have something aggregate around a big successful project.
I tend to agree with Cameron. The mutual benefits of collaboration are generally a sufficient driving force. That said, initiatives like Innocentive, where people are paid to solve problems, seem to be working. But there we're moving away from Open Science, which I assume you favor in your proposal.
What stops me and lots of other people from helping is simply that I don't have the requisite skills and/or equipment to really contribute. That seems to be the bottleneck.
If you have money and just want to outsource an analysis there are plenty of private companies who will do exactly what you ask.
One point here is that it may remain necessary that the discovery is done by people. I spend a lot of my time as a matchmaker, introducing people to the people who can solve their problems. Its not clear how 'supernodes' in a human network would be rewarded. What is the contribution to a piece of research that merits inclusion on the paper? If we move beyond papers as the metric how is that contribution measured?
Cameron,
I think the ultimate metric is the ability to get members of the establishment to write strong letters of support explaining the importance of the non-traditional work done by a particular researcher. This will come up during tenure/promotion and proposals.
Regarding the currency issue - I've gotten mixed opinions on whether there is a need for this. People outside the "open science" mold think it's crazy to NOT have it based on some kind of currency, whether it be monetary or otherwise (and a significant proportion think it should be monetary). Some of these people are scientists themselves, so it's interesting that they endorse this, but I suspect that they were also thinking in terms of how the website would be most financially successful - i.e. the business plan. I agree that mixing money into what should be a reciprocal scientific community seems a bit contradictory to the aims of open science.
I think one key requirement is that people should be able to derive some kind of reward from helping other people. Maybe this can simply be the knowledge that you contributed to someone's research. But very few people in academia have time to spend on a stranger's problems - unless they are introduced to that person or may gain something professionally. A main idea should be that what is difficult or inaccessible to one person is relatively trivial to someone else. If they somehow earn "credit" that they can then use to solicit help for themselves it may make people that much more willing to help. Does it make it less like currency if we call it "karma" instead? ;)
@Jean-Claude: related to the above, outsourcing is useful, but it doesn't necessarily build up any of that "karma", which I think is central for building community.
On collaboration: for projects where there is significant intellectual or technical contribution from both parties it makes sense to have a formal collaboration, which usually results in a published paper. What are your thoughts on very small (but still crucial) parts of projects such as acting as 1 out of 10 gold standard humans?
@Cameron on supernodes: I think matchmakers would still have a significant role in such a community - even if people posted problems and others were available to help, they will still benefit from someone who is familiar with both the problems and helpers currently out there and can bring them together. Of course, we could automate this part too through feeds and alerts, but having something like community mediators would still be very valuable. Maybe people such as yourself could even have "office hours" on the website!
In general, there are a ton of directions this could go, many of which haven't really been explored to my knowledge. What ends up being developed may not resemble this proposal at all, and what ends up working may be something else entirely. I agree that the most important thing is to have buy-in and critical mass - which is where a carefully orchestrated launch plan (perhaps similar to that of Facebook) becomes essential.
Shirley - the small units translate as goodwill. We all keep an account of it with all the people we know.
This is a great topic you started :)
Hi Shirley,
I hadn't seen this blog before but found it via FriendFeed.
I had seen your name before from reading blogs by PMR, Cameron and others.
"I spend a lot of my time as a matchmaker, introducing people to the people who can solve their problems...."
I can certainly relate to that Cameron. That was one of the main reasons I joined The Synaptic Leap. Getting more folks interested in actually doing science in the Open is one of the things that I'm trying to do.
As the numbers grow, the ripple should turn into a wave.
Re. Synaptic leap, I feel a blog post coming on.
I am with Cameron and Jean-Claude. Trying to "define" currency is not going to work. People have very different expectations, and as in other places, the moment you make things monetary from the get go, then trouble arises, since the reasons for doing a lot of interesting things change. This from an avowed capitalist.
There is something to be said for "scientific capital". This could mean the ability to publish faster or take on topics that you might not have otherwise, the mutual benefits that Cameron and Jean-Claude talk about. There is the possibility of taking an idea and building upon it, perhaps starting a company.
You need to ask the question, does community create more opportunity than it closes? In a closed system, you lose out on more things that might think.
Companies are learning this as well. Many companies had a "not invented here" mentality. It's been shown that in most cases, that just doesn't work. It's better to do what you are good at and partner with someone good at a complimentary technology. You benefit and your customers benefit.
I like the idea of scientific capital and I think it is something most researchers can appreciate. But as open scientists are still a rare breed and we need a baseline number of participating members to make any community-based website work, how do we build up the numbers?
If we build it, will they come? If not, we will need to recruit, and it's not clear how to convince busy scientists to join a new web community, much less become active participants.
Of course, it's possible this community could still benefit even if it was only made up of a dozen or so dedicated people. Someone is bound to help someone. It doesn't have to try to become huge overnight, and we don't necessarily have to go by industry metrics.
Shirley,
I think the way to get more people involved is to give them working models and help them set up. There are many people out there who would be happy to share if they just had a bit of help. See Gus Rosania's wiki for a great example.
@shwu Its interesting that you find people think there needs to be some sort of exchange currency. Is it because they are thinking of fairly large pieces of work that they don't have the resources to do? Or just that they don't trust people to return the favour?
@deepak How do think we can go about monetizing this 'capital'. Been thinking quite a lot about these things from an economic perspective. If it can be done then we can re-invest in actual resource to grease the wheels and make things happen. Good will is an illiquid commodity.
@Cameron, granted, my sample size was small (n ~ 2), and they were probably more business-minded than anything else. I think they were thinking less of their own motivations and more in terms of whether funders or VCs would buy the concept. They might be right in thinking that most investors will scoff at the idea of karma as currency, but even if that is true, the tide does seem to be changing, and social entrepreneurship is becoming more popular than ever. I think that means the probability of success is higher now than ever before, not just for for-profit socially-conscious ventures, but also for not-expressly-for-profit socially-conscious initiatives as well. This coming from someone with no background in business, of course!
I apologize if I'm taking the comments on a tangent but I'm wondering what your target audience should be.
Are you planning on only including those who have very specialized or numerous resources? I would think that those types of people might be a little difficult to win over at first. If you want a large base to build upon maybe you should look at the people who have relatively smaller resources but have a genuine motivation to contribute.
As an example, I'm thinking of perhaps "crowdsourcing" work that's slightly specialized and repetitive/voluminous to undergrads who need credit for "work in the field". It might not be paid but they can add it to their resume and say "I contributed to research done by xyz". Plus, since it's on the network you can reach tons of people who want to do the work but might not have the means to seek it out wherever they are. There are a lot of very bright people out there who are looking for a way in the door and an opportunity to contribute.
I apologize if I'm missing the point but I'd hate for you to miss out on the large crowd of "limited" resource people.
@RyanW, I am not discounting that audience at all! In fact, that was sort of the idea in my mind, anyway - to have this be a place where "grunt work" such as gold standard evaluations could be accomplished quickly and painlessly. Just as psychology departments tap into the student population for surveys and study subjects, this science exchange community could easily farm out some of its more tedious but necessary) tasks to those with less extensive or less specialized skills, or those who just want a quick little task to do to help others. The actual scientific collaborations would be the attention-getters for sure, but tasks like these might be the grease that keeps the gears turning.
Excellent, that's good. I think if you want some quick and wide exposure whenever this is rolled out that's the group to target. They're young and more apt to jump on something new, especially if it benefits them. Someone mentioned Facebook earlier, that would be a great funnel into this app, given the ties to higher education.
@Cameron ... Personally I like the idea of some form of a a futures market, which groups can sign into. That would allow people to placing small bets on projects with commercial potential and working with agencies like science commons to get the tech transfer details worked out. Then someone could run with the idea and commercialize it.
Thank you Shirley.
I have only sporadically perused the Open Science blogosphere, mostly Jean-Claude Bradley, Bora Zivkovic, and Bill Hooker. Just now discovering Cameron Neylon, who led me to you!
Your ideas about mutual outsourcing are fantastic. As for your idea for a 'credit' system, I am on the side leaning towards it. It would exploit the natural human instinct for competition (in this case, to accumulate credits), but in a manner that would align better with overall scientific/medical progress. To accumulate more 'credits' (or $$), outsourcee labs would strive to do a good job. This in turn would lead not only to more credits, but a higher public credit rating, which would attract more 'business'. Conversely, successful outsourcee labs could then spend some of that credit outsourcing parts of their own projects to other labs. Ironically, a competitive market would be established, driven by collaboration!
I would love to try to help in any way possible (as time permits!). With JCB's help, I started a collaborative science wikispaces site, where research proposals get optimized through a crowdsourcing approach. Eventually it would be great to award independent funding for cooperative execution of these proposals.
But besides attracting funding, there is a requirement for some sort of rating system. Perhaps you and others could take a look at my preliminary thoughts on such a rating system, tabbed the 'Public Contribution Rating', at http://sharescienceideas.wikispaces.com/Public+Contribution+Rating
Excited to join the OS movment,
Noam Harel
Hi Shirley, Thanks for bringing up this topic - I just read your post and liked the idea. I agree that a collaborative community/market is much needed. But I had a negative visceral reaction to motivating directly with money. I'm a member of an online network for doctors called Sermo and find myself frequently turned off by their monetary incentives to answer questions on the site (posed by other docs) or solve diagnostic dilemmas. For hefty challenges (e.g. like those posed by Innocentive...e.g. asking people to synthesize novel biomarkers, etc.), I can see rewarding people with money, their name on a patent, share in a company the evolves from their contribution, etc.. But to promote most Q&A, collaboration, etc, I think that non-monetary incentives are quite powerful and far less tainted.
Most of us want to feel like we're contributing to a cause, advancing science, and helping our peers (especially people to whom we're connected). But a community's recognition of a scientist's contribution can also be a powerful motivator (many examples in the non-science space). One could recognize scientists visibly for their contributions, including, in a quantitative way, based on credits earned or some other simple metric. You can also imagine some interesting ways to quantify the absolute and relative impact of someone's contributions...which could not only promote participation but also be an asset for advancing their career (Michael Nielsen talked about this last point recently in his blog). There are other non-monetary incentives for involving people in the community but I'm realizing this comment is getting exceedingly long :-).
Great idea again...hope you post more thoughts on this topic!
Post a Comment