science by robots
Jul. 12th, 2005 02:37 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
nibotAfter several days of fighting with the peculiarities of the computer system at LHO (LIGO Hanford Observatory), my analysis program is good to go—and it is awesome indeed. All I need to do is type "
Kent and I joked that a proper scientific analysis program should start with the raw data and ultimately typeset and mail the resulting publication. I wasn't really kidding. Read paragraph three of Background: Kepler. This time I might have it: type "
Pull the page out of the printer and read what you have discovered.
make submit" and my program finds the data to be analysed, constructs a tree containing thousands of processing jobs that will perform the analysis, and then submits these jobs to the computing cluster at LHO. Working late at night on a Sunday when nobody else is using the cluster, I have all of the computing resources to myself: 276 processors, each running at several gigahertz, each one with a gigabyte of RAM, with a collective total of twenty seven terabytes of hard disk space, and access to a robotic magnetic tape system containing who knows how many terabytes more. Amazingly all of this sits in one room in a remote, desert corner of Washington State. Amusingly the data processing program consists of only 216 lines of Matlab, 100 lines of Python, 26 lines of Makefile, with a pinch of sed and dash of other unix goodness thrown in for good measure. Programs that write programs: that's what makes CS fun. As I said: type "make submit", wait, then results.mat will be sitting there, ready for ingestion by Matlab and consumption by the analysis phase.Kent and I joked that a proper scientific analysis program should start with the raw data and ultimately typeset and mail the resulting publication. I wasn't really kidding. Read paragraph three of Background: Kepler. This time I might have it: type "
make analysis" and that results.mat file will be picked up and churned through the analysis. Hypothesis testing is done. Figures are generated, in PNG and EPS. The LaTeX typesetter is run, producing both postscript and PDF. If I were properly awesome, the proper concluding text will be subbed in given the results of the hypothesis testing. Ultimately: Type "make all." Robots in Washington State awaken, transcribe signals from magnetic tape to disk. An army of machines grinds through the data. A program in New York State performs model fitting, hypothesis testing. LaTeX turns this into a beautiful manuscript, which the Laserjet 2300dn fuses to the page.Pull the page out of the printer and read what you have discovered.
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
no subject
Date: 2005-07-12 07:19 pm (UTC)I guess the ultimate in confidence would be to have the code not only typeset your work, but post it to arxiv.org and submit to one of the Phys Rev journals without your even having to read the final product :-)
automated analysis
Date: 2005-07-12 07:33 pm (UTC)e.g. http://arxiv.org/pdf/hep-ex/0504041
If it works, I guess I'll be out of a job ;)
Re: automated analysis
Date: 2005-07-12 07:46 pm (UTC)What I wrote about above is about having monkeys make robots to do science, instead of having monkeys do the science themselves. It still requires monkeys, so we're not out of a job—the job is just one tenth as painful. Now we need to make robots to do the job of these new monkeys and then we'll have even less to do. Repeat until all the monkey has to do is type "
do science" on the computer terminal. Then us monkeys can go relax in our hammocks and wait for the issues of Science to arrive periodically.Re: automated analysis
Date: 2005-07-13 07:27 am (UTC)no subject
Date: 2005-07-12 08:09 pm (UTC)no subject
Date: 2005-07-13 07:29 am (UTC)no subject
Date: 2005-07-13 10:30 am (UTC)hello stockholm!
Date: 2005-07-13 03:23 pm (UTC)Who is this, by the way?
no subject
Date: 2005-07-14 01:37 am (UTC)