Edgar is almost as outdated as the TESS system for patents. I'm surprised someone hasn't "hacked" it long ago. Unfortunately, due to Sarbanes Oxley a lot of that infrination is required to be public.
I was just asking for an opinion from someone in that camp.
Your answer does not give faith in your business practices, as a client or as an investor. I suggest you work on communication, even with strangers on the internet.
Sympathy for the SEC on this one. For its faults, EDGAR is an elegant, useful tool for disseminating a massive amount of information. In an effort to be robust in the service of its presumably thousands of poorly-configured clients (academia, hobbyists, day traders, etc.) it's probably trivial to mistake a hostile act for poor programming practices. We've used this system successfully to fuel tons of projects and only on a few occasions been rate-limited despite numerous logic breakdowns, fork bombs, etc. resulting in traffic spikes that would have been easily mistaken by a smaller organization to be DoS attacks. While it's easy to say they should have been more vigilant, they have probably been deliberately permissive in the service of their mandate. Thanks to their patience, our projects have been served. That sort of patience probably leads to overlooking something like this.
Good catch. Spikes in option activity before the release of non-public information is the clearest indicator of insider trading. If it helps, its also trivially easy for the SEC to identify the traders involved so at the very least they are already looking for these guys to trace where they got the information.
Well, for one, it's 34 years old and is simple enough for non-technical people to use but robust enough for institutional use. For another, it's scaled to accommodate 3 decades of exponential growth in public filings and regulation. Did I mention it's 34 years old?
I'm not sure about exponential growth in public filings (and EDGAR does not track regulation, so it's not relevant). For example, the number of publicly traded companies is down about 70% compared to twenty years ago: https://fred.stlouisfed.org/series/DDOM01USA644NWDB
Could you share a link showing exponential increase in the use of EDGAR?
While there may be fewer companies, are there also fewer stock transactions, etc? This system tracks a lot of different types of transactions involving public companies.
File sizes are also increasing because filers are adding structured data[1] to their SEC filings. Also, most SEC Filings are now verbose HTML files instead of plain-text files.
And not really exponential. Something interesting happened 2003-2004 though. We can perhaps observe lead up to dot-bomb (1996-1999 growth) and the 2008 financial crisis.
It's an index - a '\n'-delimited list of strings indicating the locations of filings. The size of those filings has nothing to do with the size of the index.
EDGAR is admirable in many ways, but it's age is really showing, especially from a UI and UX perspective. I really wish the government would spend some effort in modernizing that aspect of it. BamSEC.com charges $35/month for essentially just a prettier UI, and it's so much more pleasant to use. Honestly, it's a pity the government never does M&A -- they should just buy BamSEC.
The public filings themselves are also a nightmare of hard to follow formatting.
Probably the product of (1) multiple changes in "best practices" data formats over 30 years (XML seems to be the format de jure), (2) optimizing for the filer and (3) diversity of filing data. Agreed it would be ideal if they were all similarly structured, but they have another set of stakeholders on whom they are imposing essentially a tax -- the filers themselves -- who have a legitimate gripe any time the SEC imposes a switching cost. It's easy to dismiss those gripes, but the vast majority of these filers aren't GOOG-sized. Many are very small, and those costs can be non-trivial to them.
Edit: Worth mentioning here that EDGAR's ease of use makes sites like BamSEC simple to create. Probably EDGAR's time is best spent focusing on the institutional intermediary. That allows competition to iterate on the best UX/UI for browsing these filings without imposing the cost on the filers themselves as discussed above.
> EDGAR is admirable in many ways, but it's age is really showing, especially from a UI and UX perspective
I like the bare-to-the-bones simplicity. In contrast to "better designed" portals of some other countries, I can usually point even the most technically-illiterate executive to the filing portal and expect to get something useful out the other end.
Disclaimer: I am not a lawyer. Please consult a securities lawyer before preparing and/or filing anything with the SEC.
I love EDGAR. I use it daily, and agree with the benefits of simplicity in its UX/UI. It loads quickly and you can get the info you need in seconds if you know where to look.
4. No Javascript required, no gratuitous frameworks*
5. Bulk downloads of quarterly data (FTP up until 2016)
*not to mention gratuitous iframes (BamSEC)
Thanks to #5 anyone not satisified with the "UI/UX" can take the data and go build their own "UI".
When data is tied up in layer upon layer of indirection, Javascript, iframes, enormous overstuffed URLs, binary formats (e.g. PDF, XLS), then others cannot as easily take the data and build their own interfaces.
Not a good approach for public data nor good use of public expenditure, IMO.
Do you know where it's possible to get a list of CIK #s associated with the company names. There used to be a .c file available on the sec site, but I can no longer find it.
Congress should look at the SEC obligations for disclosure (the delay seems wrong given the organization's mission of transparency... the SEC would sanction a public company for a similar failure to disclose) and why the SEC decided to wait a year to inform the public. Not disclosing the hack back in August 2016 strikes me as a political decision by the Obama Administration to avoid criticism during the home stretch of the election.
If it was intentionally covered up by Obama's administration, why did Trump's administration wait 9 months to make it public? Not everything has to be politically motivated.
Probably both. That's a pretty good-sized arbitrage opportunity.
Think about the number of people in the financial world who are looking to get a percentage of a second advantage with high-speed trading. Hard to believe they'd pass this up if they knew about it.
> The Securities and Exchange Commission says its corporate filing system was hacked last year and the intruders may have used the nonpublic information they obtained to profit illegally.
"The U.S. Department of Homeland Security detected five “critical” cyber security weaknesses on the Securities and Exchange Commission’s computers as of January"
"The report’s findings raise fresh questions about a 2016 cyber breach into the SEC’s corporate filing system known as “EDGAR.”"
"... it shows that even after the SEC says it patched “promptly” the software vulnerability after the 2016 hack, critical vulnerabilities still plagued the regulator’s systems."
As much as the UI appears dated, Edgar is consistently one of the fastest websites I use on a daily basis. Punch in a ticker under "Fast Search" here (https://www.sec.gov/edgar/searchedgar/companysearch.html) and click around the filings for a bit to see what I mean.
How come SEC only discloses this now? Isn't preventing insider trading a goal of financial regulations? If so, is there a reason SEC didn't disclose it as soon as they gained knowledge of the leak?
I think this goes to show that any data collection is dangerous, even if government thinks they're the good guys (when in fact they're naively collecting data that could be weaponized against them very easily). The EU is on to it as well with MiFID II granular reporting of financial market transactions starting effectively next year.
Though I'm not sure the Brits are going to play along (where the majority of trading happens in Europe); after all, why should they go through the trouble of implementing MiFID/MiFIR when leaving EU anyway?
Well, just because this was disclosed to the public now doesn't mean it wasn't previously disclosed to Treasury, FINRA, the US Equity SROs, and the ESMA. The surveillance systems surrounding almost any market center are substantial, and can seek the sort of behavior having inside information might appear as.
That makes sense, but do we have info on whether that actually happened? Who's overseeing the overseers? The only theory under which it would make sense to withhold that info I can think of are criminal investigation tactics, but there's no actual info on suspect transactions, is there?
Broadly and vaguely trusting in surveillance isn't convincing at all. The usefulness of surveillance must be weighed and re-evaluated against risks, as demonstrated by this incident.
I guess by surveillance I was speaking purely transactional surveillance, and typically investigations into transactional behavior takes time, and under certain circumstances would result in criminal prosecution (on top of any regulatory fines/punishment).
It also takes time to coordinate investigating multiple different asset class transactions, in the event, say, equity derivatives are used.
And no, we don’t have info on what happened and what was taken, and certainly nothing about any regulatory investigations.
61 comments
[ 0.26 ms ] story [ 139 ms ] threadhttps://www.sec.gov/news/public-statement/statement-clayton-...
To be fair, I've used EDGAR and it is <cough> very legacy. So no question it was going to be completely compromised.
I'd say the mistake was putting non-public information on it in the first place. The risk assessment for private data exposure was extreme.
the united states has protected trade secrets by law far longer than its been on the transparency train.
there are reasons for that, and the corporate disclosure requirements has had no effect on creating a "fair market"
Your answer does not give faith in your business practices, as a client or as an investor. I suggest you work on communication, even with strangers on the internet.
Btw, hackers accessing this info could possibly explain something I tweeted to them about 10 days ago:
https://twitter.com/jeffallen6767/status/907171360769662976
Could you share a link showing exponential increase in the use of EDGAR?
Index - Q1 1994 - 3mb - https://www.sec.gov/Archives/edgar/full-index/1994/QTR1/form...
Index - Q1 2017 - 45mb - https://www.sec.gov/Archives/edgar/full-index/2017/QTR1/form...
[1] https://www.sec.gov/structureddata/what-is-structured-data
http://hpics.li/2e78909
[1] https://www.sec.gov/info/edgar/regoverview.htm
The public filings themselves are also a nightmare of hard to follow formatting.
Disclosure: Last10K.com is a side project of mine
Edit: Worth mentioning here that EDGAR's ease of use makes sites like BamSEC simple to create. Probably EDGAR's time is best spent focusing on the institutional intermediary. That allows competition to iterate on the best UX/UI for browsing these filings without imposing the cost on the filers themselves as discussed above.
I like the bare-to-the-bones simplicity. In contrast to "better designed" portals of some other countries, I can usually point even the most technically-illiterate executive to the filing portal and expect to get something useful out the other end.
Disclaimer: I am not a lawyer. Please consult a securities lawyer before preparing and/or filing anything with the SEC.
2. Multiple open data formats, including TXT
3. Easy-to-follow organization (CIK), minimal indirection
4. No Javascript required, no gratuitous frameworks*
5. Bulk downloads of quarterly data (FTP up until 2016)
*not to mention gratuitous iframes (BamSEC)
Thanks to #5 anyone not satisified with the "UI/UX" can take the data and go build their own "UI".
When data is tied up in layer upon layer of indirection, Javascript, iframes, enormous overstuffed URLs, binary formats (e.g. PDF, XLS), then others cannot as easily take the data and build their own interfaces.
Not a good approach for public data nor good use of public expenditure, IMO.
I am grateful to Carl Malamud for creating EDGAR.
Think about the number of people in the financial world who are looking to get a percentage of a second advantage with high-speed trading. Hard to believe they'd pass this up if they knew about it.
https://www.nytimes.com/aponline/2017/09/21/us/ap-us-sec-cyb...
> The Securities and Exchange Commission says its corporate filing system was hacked last year and the intruders may have used the nonpublic information they obtained to profit illegally.
This is the most problematic part.
"The report’s findings raise fresh questions about a 2016 cyber breach into the SEC’s corporate filing system known as “EDGAR.”"
"... it shows that even after the SEC says it patched “promptly” the software vulnerability after the 2016 hack, critical vulnerabilities still plagued the regulator’s systems."
http://www.reuters.com/article/us-sec-cyber-weaknesses-exclu...
I think this goes to show that any data collection is dangerous, even if government thinks they're the good guys (when in fact they're naively collecting data that could be weaponized against them very easily). The EU is on to it as well with MiFID II granular reporting of financial market transactions starting effectively next year.
Though I'm not sure the Brits are going to play along (where the majority of trading happens in Europe); after all, why should they go through the trouble of implementing MiFID/MiFIR when leaving EU anyway?
Edit: preventing insider trading
Broadly and vaguely trusting in surveillance isn't convincing at all. The usefulness of surveillance must be weighed and re-evaluated against risks, as demonstrated by this incident.
It also takes time to coordinate investigating multiple different asset class transactions, in the event, say, equity derivatives are used.
And no, we don’t have info on what happened and what was taken, and certainly nothing about any regulatory investigations.
https://www.facebook.com/l.php?u=https://www.wsj.com/article...