Promethease[edit]

is a wiki expert system. New bug reports are welcome below, closed bugs are in the page history.

Dangerous Download?[edit]

As of August 2015, chrome has started to warn that it might be dangerous to download this .zip file. While we strongly encourage careful computing, this is a false alarm in chrome. You can see our attempts to resolve this at https://productforums.google.com/forum/?#!msg/webmasters/tGYs1sBbefE/_xG01EGYbOcJ. You can use https://www.virustotal.com/en/ to check your own download however doing so publicizes your genetic information and your IP address, which you might consider a larger security concern. There is no .exe or other dangerous file format inside our .zip, and for the moment there seems to be nothing we can do to convince google of that.

Fails to Open Some Report Sections[edit]

This is the most common 'bug' report, but it is something only the user can solve.

Try watching the video starting at 6:44.

Promethease gave you a .zip file. You need to open that .zip file up, but instead windows has allowed you to look inside the .zip file. There are other important files inside the .zip, but since you didn't actually unzip the file, you are unable to see them. Find the file in your downloads folder

Then really unzip the folder.

http://www.wikihow.com/Extract-Files

On Windows 7-zip is free and winzip is easy.

iPad

Updated Reports[edit]

I noticed that when we created a Promethease report for my brother it was a newer version than mine (created earlier this year: Version 0.1.53). Do I have to pay another $5.00 to get this version or is there some way to upgrade?

A report is a snapshot in time. New scientific literature is published daily, and users continue to improve the wiki summaries. To give you up to date information we need to rerun the process. This is why you have a download, which you can keep for as long as you like. If you decide you want fresh information in a month or a year rerunning is up to you, and you do need to pay for the newer report.

Actually, you can rerun Promethease for free by downloading the EXE file and running it yourself. It will take several hours to finish (it has to query every page individually), so just leave it running in the background while you do something else. You will get a non-interactive report, but it includes all the up-to-date information. CarlKenner (talk) 09:38, 12 October 2013 (UTC)

General[edit]

My Report won't open[edit]

I have a report of over 10MB which won't open. When I first open the report, only a ‘black’ Active X alert if present (CPU running 100%), then the report screen is presented with the Active X alert ‘white’, I click the alert bar and click Yes to the question about running Active X, get a continuous hour glass with 100% CPU operation.

Plain old IE should have no problem with it. But perhaps you've got layers of antivirus (and or viruses) battling it out. Please try the same file in another browser. I'd suggest chrome or firefox

Recognized file formats[edit]

Promethease knows how to read these specific formats:

23andMe
Ancestry.com
FamilyTreeDNA (the SNP test, not the STR test)
NatGeo
DNA.land
Complete Genomics
- The dbSNPAnnotated.bz2 file is the most useful
- masterVarBeta files are also supported
Affymetrix GCOS
VCF (if the ID column has dbSNP rs#s)
Illumina GenomeStudio
WeGene
many more, including those from companies that have come and gone, like:
- deCODEme
- Navigenics

Do not attempt to uncompress or convert between formats. Many of the conversion tools do not work well and can do harm. Just upload your original untouched data. If you cooked up your own file you may end up with a warning about it being unrecognized. Since all of the formats from the various companies are just tab or csv delimited, promethease relies on these headers to know which format to expect.

Is Promethease Open Source?[edit]

I was wanting to play around with the promethease code to see if I could make some improvements, but was unsure if promethease was open source. Specifically I wanted to change the way the report is generated. While the report is very detailed, I feel that normal consumers would find it difficult to properly browse the report. Further division and graphical representation of the data would be highly beneficial. For example, division of medical topics by area, color coding records to represent problems or benefits (similar to magnitude) and symptom cross checking (user types in symptoms which are compared to conditions for which he/she has a genetic predisposition)

No, Promethease is not open source. However there are several parts to Promethease, perhaps what you're looking for is available. The wizard interface that asks for your genome file and makes the html report ... that is closed source. But the report that it makes is pure html + javascript. The newer ui2 reports have a clean separation between the data and the code. Do a view source on

http://files.snpedia.com/reports/promethease_data/genome_Lilly_Mendel_Mom__20080505171156_ui2.html

If the improvements you have in mind can be done via javascript/css let's discuss your changes and I can add them to the next release.

The division of Medical topics can probably accomplished purely via making edits to Category:Is_a_medical_condition. Some of the things you've suggested are more dependent on data which is not yet captured in SNPedia. If you're interested in providing that data, I can arrange new fields such as Magnitude, Benefits, Problems very easily. Until there is a significant collection of real data, Promethease can't help.

Who owns Promethease? Would it be possible to make it open source?

Promethease is owned by River Road Bio and developed by User:Cariaso and User:Lennon as a closed source application. The Terms of Use page has more related information.

What's new in Current Release[edit]

The notes below are far from comprehensive, but do point out some of the most notable differences in some releases.

0.1.161

free reports now show the summary for genosets

0.1.157

fixes a problem with ui2 reports not showing the search box correctly, due to a content delivery network that decided to stop serving bootstrap

0.1.156

more touchscreen friendly ui2 reports
conflicts report knows how to deep links into ui2

0.1.155

filter no-calls from 23andMe & decodeme data, improved detection of conflicting genotypes

0.1.154

too many open filehandles during F1 of files from different companies

0.1.153

linux screen redraw,
ui2 bootstrap
fixed the link to the main report when run under windows
Ancestry.com file format

0.1.152

Improved handling of Unicode in filenames
Gene reports

0.1.151

Recompiling with latest version of a library dependency so that the downloaded cache file works consistently on all platforms

0.1.150

NatGeo has decided to change their format, in a way that we cannot support. This release fixes some handling of files with spaces in their names, and a new binary format for the paid download snapshot.

0.1.149

NatGeo support

0.1.148

better Graphs for each Medical condition, Topic, etc. Example

0.1.147

Graphs for each Medical condition, Topic, etc. Example

0.1.146

Genoset chaining bug fix

0.1.145

Family Trio related

0.1.144

Linux native

0.1.140

WINE mostly supported again

0.1.125-139

Hundreds of little fixes, new features, tweaks, etc

0.1.124

UI2 improvements that move the Magnitude and References sliders into the blue box.

0.1.123

uses Repute and introduces the Good & Bad report sections into UI2

0.1.122

uses Repute and introduces the Good & Bad report sections

0.1.119, 0.1.118

more network error tolerant, thanks to Axel Pettinger

0.1.117

restore the 'complicated' report section

0.1.116

improves the retry behavior of network requests when SNPedia is overloaded
fixed a bug which prevented the ability to change the outfile destination
Thanks to Greg Goddard for reporting both

Minor version upgrades ?[edit]

My question: I'm sure there are many things that occasion the creation of a new version. But do most version upgrades (that have no description under "What's new in Current Release") -- say from 0.1.145 to 0.1.146 -- typically contain at least few new bits of info on some SNPs? Or are they typically about other sorts of tweaks, involving stability for ex.?

Obviously, I ask this as an obsessively genetically self-interested person! :)

Seven7 (talk) 13:32, 17 October 2012 (UTC)

The section above was at one point an attempt to do that. But my releases don't often merit an announcement. I'm on the 146th public release, with over 2700 commits in the codebase. Most releases are incremental improvements, often to systems you never see such as the User:SNPediaBot or the server administration code. 0.1.146 was triggered by a significant fix to processing of genosets, and there is a hint of this in the edit history on Oct 16th. That history also hints at other fixes. 0.1.145 was mostly about a linux release, but was also important for fixing a problem with the download of the cache during paid runs. 0.1.144 had improvements to trios developed while working on the Clarity Challenge. In the past I've found, when I announce a new feature it tends to have some sort of bug, and I end up having to rush a fix out. If I quietly slip them out, there seems to be a better track record. Since the code is still maintained by a single person, there isn't much need for meetings or planning documents, so a What's New is just extra work and really I've got plenty as is. Since version 0.1.67 (3.5 years ago!) it checks on startup and warns you if there is a newer release. Someday I hope that it will auto-update, but so far that feature isn't ready. Until that time, Promethease is steadily improving, and newer releases are better, so upgrade. Even when you've not upgraded, SNPedia is continuously improving, and even runs just a few minutes apart can sometimes show improvements in the content.

TL;DR. Upgrades are better, but I'm unlikely to detail what's new with each release. --- cariaso 23:58, 17 October 2012 (UTC)

I can't get a successful promethease creation run[edit]

I'm using Windows 7, and I read that the fast version doesn't work with Win7, so I first tried the free option. I loaded the 24GB 23andme txt file of my data, and it seemed to accept that OK. It downloaded a few batches of SNPs, but then it started downloading each SNP separately, about one per second. In hopes that it might go back to downloading SNPs in batches, I let it run overnight. This morning (About 12 hours later) it was still downloading individual SNPs. Since there are 960,613 SNPs in my 23andme file, I calculate that at one second per SNP, it will take 266 hours to download the whole list. So I cancelled out of the process and tried the $2 version. The Amazon cash transfer worked without a problem, although my Amazon password had to be changed because my original password was too short by current Amazon standards. Then the speedy download started OK. However, as predicted, it hung up at about the 80% mark on the download, so that didn't work. So I tried the free version again, and again, after a few minutes of downloading SNPs in groups, it reverted to downloading one SNP at a time.

I tried to find the logfile that the program said would have the details, but there was no logfile in the temporary location specified. There was a lot of promethease files there, but no logfile.

I really want to use promethease with my 23andme data, but there seems to be no practical way to get it to work.

After further investigation of what the program is doing line-by-line (based on repeated runs and screen captures to create my own logfile, I now see that promethease is only downloading the 34,931 SNPs in the snpedia database. At one second per SNP, this would only take 9.7 hours to complete, so maybe my overnight run was almost complete when I terminated it this morning (my estimate of one second per SNP is very rough and based on one spot check, and it could have varied during the run. My download speed this morning was 1.32 Mbps (using DSL), so maybe that explains why it will take 9 or more hours rather than the 4-hour estimate that is quoted.

Is this an accurate picture of what is happening, and should I try another overnight run tonight, or is there some other solution? And does the fast version give the same results as the slow version?

Your summary seems quite accurate. Things may have been running a bit slow in the last few days, as we've been hit pretty hard by some clumsy 3rd party bots. The differences between Paid and Free is detailed at Promethease/Features. The pattern for who succeeds / fails on paid runs remains unclear, but I suspect retrying a paid run will reproduce the same failure. Running from another computer is more likely to work. If that's not an option, another overnight free run seems viable. The speed of your internet connection is unlikely to have had an impact. You remain anonymous to me, so you'll have to email info@snpedia.com to request a refund for the failed paid run. --- cariaso 21:59, 5 October 2012 (UTC)

I would be ashamed to have an application as poorly performing as this one free or not. It's pointless to waste the bandwidth to even try it.

We got hit by a bug for a few hours today when I messed up a server config. If you were seeing '503 snpedia seems busy' that was the problem, and it's now fixed. If you're just complaining for other reasons, either provide some useful details, or continue to vent anonymously.

"Edit mode"?[edit]

I'd love to be able to help improve Promethease. While I learn my way around, and learn (much...) more about genetics, one thing I could do is make little language improvements -- fixing typos, etc. I thought I'd be able to do this in edit mode when I'm looking at a Prom. run. But nothing seems to happen. I assumed some sort of Javascript window would open, but I see no changes when I select "edit mode". Have tried in both Firefox and IE (latest versions of both, Windows 7). ----User:Epsilon4 2013-01-11 18:54 (UTC) 18:54, 11 January 2013 (UTC)

Throughout your report there are links like rs1234 (A;C) where the rs# and the genotype link to different pages. By clicking the 'editor mode' checkbox, most of those links will now point directly at the edit page, not the reading page. The report does not (yet?) offer any editing other than what you would find when you click on the 'edit' or 'edit with form' links inside SNPedia. --- cariaso 19:02, 11 January 2013 (UTC)

OK, thanks for the info! (Note to self, again: watchout for the SNPTips Bug before contributing to the wiki.... Hope they fix that soon.) --User:Epsilon4 2013-01-11 20:02 (UTC) 20:02, 11 January 2013 (UTC)

FASTQ format / exome dnadtc.com[edit]

Can Promethease read this format? Would the report be more useful (extensive) than from 23andme com? If Promethease can not read this, are there conversion tools available?

Promethease is unable to support FASTQ, but DNADTC also provides VCF files which Promethease understands.

Why Administrator required -- Windows promethease 0.1.157[edit]

What is the reason for requiring the Windows version of promethease to run as an Administrator? Being closed source this seems highly suspicious.

Because I'm a terrible programmer. So use the web based version. --- cariaso 00:23, 8 December 2013 (UTC)

Desktop[edit]

These questions are only relevant to the Desktop version, not the web based one.

Genoset summary doesn't show up in free reports[edit]

The genoset summary should be at the start of each box in the free report, like it is for SNPs, but it isn't there at all. It is in the UI2 version though. That makes it very difficult to write good genoset pages because everything has to be repeated twice without sounding like it's being repeated twice for UI2 users. And it would be really nice if the summaries were in bold in the free report like they are in the paid report. CarlKenner (talk) 20:22, 6 December 2013 (UTC)

Extra process[edit]

It seems like running promethease causes another process to start running called nsB50A.tmp (or any other 4 hex digits after the ns). This is typical of various rootkit/worm/trojans. Any idea?

No virus, just part of the mechanism for making the application simple. It needs many DLLs and support files, but I don't want people to have to deal with an installer, especially for these earliest versions. I like single file .exe apps such as PuTTY. The single file .exe is actually a NSIS installer. When you run the app, that unzips the necessary support files into a temp directory and launches the UI. When you click the 'begin analysis' button a second thread begins. This second process does the very long running communication with the SNPedia.com server (which will often block due to network speed/unreliability) to run without interrupting the UI thread which needs to remain responsive. I'd encourage you to run it through mitmproxy to observe the traffic and watch how truely boring it is. If you'd like to study the mechanism more closely, a fork of the relevant code is released at https://bitbucket.org/cariaso/runblast as an open source project.

Fails and then requires payment again?[edit]

Promethease failed after creating most of a report. When trying to run again in the hopes it worked this time, it then asked me again to pay again to speed it up and get the features I used.

I tried to continue again without paying again, but it didn't give me the same inheritance option for my spouse I used the first time when I paid.

The promethease.log was overwritten on re-startup so I don't have useful info for you, but it did get pretty far. The callstack it showed mentioned interaction with my 23andme primary ZIP file, a mention of file detection, and finally the StopIteration exception.

I'm happy to pay another time if needed, but if it fails again I'd be really depressed.

Any way to get it to work with the advanced spouse option?
Any way to avoid having to pay again? (Amazon transaction ID: 14QI8O88K743ET1DJGSMZPP9LP98FQMN5PL)
Any info I can provide to help you further?

This is now refunded. When reporting a bug it is helpful to know

Amazon transaction ID - Only if you paid
The name Amazon.com knows you by - Only if you paid. (Easier for me to see than the transaction ID)
Information from promethease.log (future versions will make this easier to find)
Your Promethease version number (it should look like '0.1.76' or similar)

Running again is unlikely to make it work. Instead email bugs@promethease.com and perhaps the next version will be able to fix the issue affecting you (and probably many other people). Cariaso 22:10, 6 February 2010 (UTC)

Length of time estimate for Free is wrong[edit]

I am assuming that this is a victim of success. On a modern Macbook Pro, I have been running processing of a 23andMe dataset for about 12 hours. It is still chugging through, but I believe your free vs paid estimate is most likely wrong. (Also, being able resume a processing effort would be great - the mac UI allows you to hit escape and close the window, losing the processing. Is the Windows or Linux versions suffering in the same way?

It has very little to do with the number of users, and is mostly due to the growth in the number of known SNPs. It is certainly true I have not updated this estimate in quite a while, and I welcome a confirmation for others in how long it takes you. Also be aware, that the microarray testing platform is an important factor. The 23andMe chip has 2x as many SNPs in SNPedia as a FamilyTreeDNA so making a run takes 2x as long. --- cariaso 02:10, 20 February 2014 (UTC)

Dear Cariaso,

I just ran the Promethease Mac version for ~16h on the current 23andme dataset. Hope this helps. Also I would like to buy the 5USD, but since I'm abroad using credit cards is always a little bit of an issue. It would be great if you could get the Amazon payment process working again, or offer Bitcoins or PayPal as an alternative payment option. Keep up the good work Thanks, Karl. --- Karl, 21 February 2014

Ok, text on the Promethease page has been updated. I want Amazon fixed, Paypal, and Bitcoins support. One of them is beyond my control. I see tremendous appeal in Bitcoin as a truely global payment option, however the significant lagtime in getting payment confirmation has kept this from bubbling to the top of my todo list. It was a stroke of luck that stripe's announcement of 135 new currencies finally prompted me to get that working just days before the amazon issues occured, and I'm disappointed to hear that those 135 new currencies still do not solve your needs. Paypal has always been rumored to be a rather hairy API, but having now gotten support for 2 other payment processors working, I expect I may be able to add it without too much trouble. What I really need to do, is to get a cohrent way for you to upload multiple people, and unambigously label the family relationships, so that from the webui you can do the family based reports. All of this is on the todo list and your feedback and payments help me to prioritize and deliver. I was getting excited for adding medical reports ala the 1.0 UI, into UI2, but maybe I should try to deliver BitCoin or paypal sooner. Comments below to suggest what you want are most welcome. --- cariaso 15:08, 21 February 2014 (UTC)

Any chance we could get Paypal added? Some of us don't have credit cards! --- Lilstar

Many newcomers due to 23andme interpretation discontinuation[edit]

Listen up:

23andme won't be providing interpretation any more. Facebook posts on their page point people to promethease and snpedia.

Here is what a non-technical person sees when they google "snpedia" or "promethease" and visit either the homepage or the promethease page: completely helpless jargon.

Yes this is a wiki but is is also perhaps one of the most important resources on the planet right now.

Someone MUST rewrite both the homepage as well as the promethease page to be more welcoming and accessible to people who think they have no hope any more in receiving any type of useful information from these services. People DON'T WANT a refund from 23andme, they want their data, and it is up to the wording on these pages to either make them understand they will get more data than they bargained for, or to just confuse them.

Pending this the biggest influx of users to promethease will likely be 23andme customers, it is up to this organization to make it work efficiently. (unsigned post by anonymous user)

I noticed the same thing. But I'm starting working on improving the quality of the 23andMe section of the health reports that Promethease generates first, because they need some work to be the same quality 23andMe users are used to. It's a wiki, so feel free to sign up and fix anything that isn't good enough. CarlKenner (talk) 20:22, 6 December 2013 (UTC)

Anonymous user "23andme won't be providing interpretation any more". That will be true for a while, but I don't expect it will be true for very long. But for the moment, you are correct. I don't disagree with your statements (except perhaps to humbly disagree with "on the planet"), but the issue is simply that we lack the number of ACTIVE users to reach the quality and tone that you're looking for. The solution is simple. Be Bold. With a large enough pool of active users, we'd outpace the FDA, the NEJM and my own hopes for providing helpful and correct information to a global (and multilingual) audience. Which is why I commend User:CarlKenner for picking up a shovel and getting to work to make it the resource he thinks it should be.

In particular, he has pointed out a bug report with the free reports, the lack of a summary for genosets. This reflects the fact that paid reports have driven the development of new features, and once upon a time genosets were a new feature. Like many other features developed for UI2 (ie, Images, Repute, ...), the time has come to propagate the improvement back to the free reports. So I will begin work on that. However I'm currently doing volunteer computer teaching at schools near Mae_La_refugee_camp and working over a satellite connection while keeping a rather full schedule of teaching, plus work for my real employer keygene.

Trust that fixing that bug in free reports is now ~~in progress~~ resolved, but it is a good reason to support Promethease with your payments, to help me continue and expand development. Genosets are only one of many many features on my roadmap. Until that time, improvements to the text of SNPedia are the best way to help the broadest audience. Even the front page is fair game. You can't break the wiki, and if we feel your text doesn't help, or misses an important detail we can easily rollback or iterate towards consensus.

SNPedia has been active since before 23andMe had a website. What is here today is here because of the efforts of an incredibly dedicated, but small core of contributors. My deepest thanks to anyone who has ever made an edit, and especially the Special:ActiveUsers.

--- cariaso 00:13, 7 December 2013 (UTC)

There's no hurry with the bug fix. There aren't that many genosets, and we can make sure the information is in the text body. So you can keep teaching refugees while genetically engineering new crops for them. But let us know if you find any interesting new haplogroups over there.

Unfortunately, I am able to temporarily break the Wiki a little, or at least the Promethease reports generated from it, so I often need to run Promethease for testing, and I can't afford to pay $5 every time I want to test my changes. Testing also takes a very long time, so I have to edit the raw data to only include a subset of the genes (eg. the 23andMe reported SNPs or the entire mitochondrial dna), and even then it still takes quite a while.

Anyway, I encourage everyone to have a go at improving the Wiki to make it more friendly to people coming from 23andMe. It's just like editing Wikipedia. Particularly the SNPs listed on the 23andMe/SNPs page. You can click on each rs number and then click on the "23andMe all" link in the box on the right to see all of what 23andMe tells users about that SNP, and then make sure that information is included (in your own words) in each of the three letter combinations in the box on the right of the SNPedia page. Also make sure they all have a magnitude greater than zero and a repute set to good or bad if they are good or bad (a few will be neutral). The summaries and text need to be concise, and it is good to put a full stop (period) at the end of the summary. Diseases or medicine topics should be made into wikilinks in the main page for that SNP, so they end up in the right categories in the Promethease report.

The FDA seems to be completely out of their depth and irrational about 23andMe and genetic testing, so I expect it will actually take a long time to reach an agreement. The FDA want us to pay $3000 per gene, with 10 years and billions of dollars of clinical trials before each update, and only the doctors would have access to the information. That's the mindset they are coming from. 23andMe's mindset is more like Promethease's. So an agreement will be difficult. CarlKenner (talk) 07:25, 7 December 2013 (UTC)

One thing 23andme did well was compiling effects from many studies together into one risk estimate. Would it be possible for SNPedia to add a section to add effect size and sample size information so Promethease could pull that information into a type of meta analysis for risk estimates? 184.57.16.174

I made a "genoset" that sort of does that for Atrial Fibrillation. But I can only add the probabilities that someone in my family or Lily Mendel's family has, and it varies by gender. So many combinations are missing. Unless we can work out their formula. But you're right, it would be best if Promethease had the ability to calculate odds itself. CarlKenner (talk) 12:51, 9 December 2013 (UTC)

Taking individual studies and blending them into one risk estimate can be a bit like chucking the individual dishes of a fine meal into a blender and claiming the result is equivalent. For Promethease to do a meta-analysis properly would require knowing that many things are equivalently controlled in all studies to be combined (including the obvious things like age and gender), and then it would have to choose which meta-analysis method to use (Random effects? Binary effects?). Our preference to date is to instead cite the results of published meta-analyses whenever possible, and if you know of such studies for any given condition that aren't already being used in SNPedia, either edit the corresponding pages yourself or call our attention to it. --- lennon

How does Promethease handle missing SNPs?[edit]

When a SNP is missing from an uploaded data set, how does Promethease (desktop version) calculate whether criteria are satisfied or not? For example, 23andMe does not include rs1799883 in its latest assay. Criteria for Gs283, Gs284, and Gs285 involve rs1799883. Somehow I'm satisfying criteria for Gs285 without having uploaded rs1799883.

Does Promethease evaluate every permutation of the missing SNP (i.e., AA/AG/GG) to see if all three variants satisfy the criteria?

no

Does Promethease assume the SNP is the variant with the greatest frequency in the indicated population?

no

I hope this question is of general interest! I couldn't find the answer in the various FAQ/information pages for the software.

it evaluates a missing snp as false. However it is possible to be gs285 without rs1799883. Part of the criteria is

and(or(gs281, gs282), not(or(rs1799883(C),rs1799883(G))), rs1042714(G))

the part

or(rs1799883(C),rs1799883(G))

is asking, if rs1799883 was a G, but being tolerant of reverse orientation. in your case no genotype was reported, so that will be False.

and(or(gs281, gs282), not(False), rs1042714(G))

is the same as

and(or(gs281, gs282), True, rs1042714(G))