- I was afraid it would change the language culture
- It was not comfortable to use
- The idea and design was, as you said, bizarre
- Use use a dynamic language, why the hell would I want that?
Years after the fact, I'm really happy about it.
First, most of my Python code don't use them, so Python stays python. But if I ever need them, there are here.
Of course, you could argue I should just use a language that has been designed for that since I need type. But that's missing the point: I don't usually start using hints. I code regular Python, because it's awesome. I just sometimes add type hints after the fact as a bonus.
The ability to do that is fantastic, even if the type system is far from perfect.
I want to code with Python most of the time anyway. I personally have very few use cases for another language, doing mostly web stuff.
Type hints are still very verbose, and convoluted, although it will get better with 3.9 thanks to 2 PEP targeting type hints ergonomics. So sure, it's never going to fit perfectly, but I'm glad it's here.
Now for a JSON schema, I guess it's the same deal. You use JSON because it fits your use case, and one day you realize your situation could now benefit from more robustness, and you add it to the mix.
Again, you could argue you should have though of that at the beginning, but projects don't have frozen requirements, they evolve. And often, you want to start lean, because it will most probably stays that way, and otherwise would be so costly the project would never grow to a point you would need robustness.
Now in this particular case, this lib is not just checking type, but also arbitrary logic. And making sure data is consistent is necessary no matter how dynamic or static your language is. There is a limit to the constraints your type system can represent.
I led a project for validating customer-provided data against a json schema. The structure of the data was moderately complex, and it was getting bound to pojos by a pretty lenient binder. This caused issues when making additions to the API since customers were sending arbitrary fields, and duck typing the data wasn't that reliable. The easiest thing to validate before binding with a json schema.
What's different this and Python type hints is that this was a public-facing API, so you have far less control over what data you'll see, and the server code was Java, so getting things strongly typed and validated early on makes your life easier.
Schema validation is more than type checking, and has been very well solved by libraries like marshmallow and pydantic for a long time. I don't think the comparison holds.
I've worked with Python professionally for 3-5 years, but only type-hinted Python for ~5 months. This is coming from someone who prefers Java, but I don't like Python type hints because they were bolted onto the language worse than classes, type hierarchies aren't mature, and if I wanted types, I'd be using Java. That said, you do need strong typing for a project once it hits a certain line count or developer count.
Java's type system is weaker than what Python's typing provides. If you want to make fun of Python's type system, compare it to TypeScript, ocaml or haskell. Python's typing is actually useful even if it's bolted on and the typeshed is incomplete.
There are languages though like eg: Groovy where gradual typing is more designed in. It's unfortunate to me that Python has cultivated this attititude because it's like local maxima that prevents the mainstream getting to soemthing better. Python is the new perl / PHP etc.
I mean the attitude that Python is "good enough" because it has type hinting and therefore it is not worth bothering with exploring other languages, especially ones that have wholly more powerful type systems. I have found (personal experience only here) that it can be tremendously hard to get Python programmers to learn and embrace other languages and approaches to software.
that's true of any sufficiently popular tech stack. some people get either way too attached to their beloved language (...do carpenters fall in love with hammers?) or just are too lazy to learn something else - assuming that programming is their profession and not just a means to an end.
This is orthogonal to type safety. In fact, schema-based validation is an excellent way to perform type narrowing even in statically typed languages. An extremely prolific example of this is parsing incoming JSON from network requests :)
This has little to do with dynamically typed languages, "JS" in the name notwithstanding. JSON is a serialisation format that is used in both statically & dynamically-typed languages. It makes as much sense to call XML validators bizarre as it does to pin this project as a weird byproduct of dynamically typed languages.
If your program produces or consumes JSON _and_ you think it would be a good idea to ensure said JSON is what you are/a client is expecting, then you need something like this in any implementation language.
what I completely don't understand is the current fad on everything-should-be-yaml where yaml is marginally easier to write than XML, a tiny bit more than marginally easier to read (caveat emptor - Norway has a country code) and otherwise exactly as broken. (i'm talking about the 'safe' variant.)
I don't think you understand static typing and you might be confusing dynamic typing with duck typing or other concepts. The presence of types doesn't mean static typing. Python, for example, is strongly typed, but it's dynamic because the type of any object can't be determined until run time. But you can still check types!
> What statically typed language lets me define a type as “number between -180 and +180”, or “string which contains only alphanumeric chars”?
Pretty much all of them. Any simple predicate like this can be encoded with witness types.
Here's an example in Java, which is hardly the paragon of static typing (i.e. it's no Haskell/Idris/Agda/Rust/Typescript):
class AlphaNumericString {
private final String str;
// use a fallible factory with a `private` constructor if you're
// morally opposed to exceptions
public AlphaNumericString(String str) throws AlphaNumericException {
if (!str.matches("^[a-zA-Z0-9]*$")) {
throw new AlphaNumericException();
}
this.str = str;
}
private static class AlphaNumericException extends Exception {
}
}
Now code can freely use `AlphaNumericString` and be guaranteed that it has been validated.
You may object and say that newtype wrapping is cumbersome but:
1. That's an argument about sugar and ergonomics, not about the semantics that the static type system enforces
3. The `AlphaNumericString` is describing a smaller set of values than `String`. In general, you should be strongly considering the methods you allow and make sure that all paths continue to enforce the semantics you intend.
Very neat. I had a similar need recently which led to a similar solution[^1]. Notably, I used a syntax nearly identical to yours, albeit without the very handy pluggable validators yours has. Nice job.
What led you to choose “!” as the “optional” modifier? My intuition would have guessed that character to have the opposite meaning.
A long time ago, before MyPy, I did a python schema checker for JSON. It grew into argument checking too: https://pypi.org/project/obiwan/
Simple stuff like:
Person = { "name": str, "age": int }
The basic idea was to use a file full of obiwan types to document the rest api. The schema was both human-readable and valid python. The api could then trivially check the schema. Once the schema is checked, of course, its safe to go walk the json knowing that things aren't going to crash on you etc.
It worked great. Still works great. I still prefer the syntax to that of MyPy.
No real adoption, of course :) But fun to remember what could have been :)
I do not think so.
If you check the example schema, it is very hard to understand it. What does ':lat' mean? is it a string or a number?
What does '!b' mean, can it only be false?
Yes, reading the docs explains what these keywords mean. But by just looking at the schema it is impossible to understand what that is. And if you check the definition of clean code which is something like "intuitive understandable" then it comes clear that this is not clean.
Strings make it easy to define as configs and can be implemented uniformly across different languages. In world of microservices, a Python and a Go service can easily share the schema definitions via some config files. Also strings provided the best minimalist syntax.
Json schema does scale down pretty well (indeed as the link above shows the simplest possible Json schema is just "{}") - it also has pretty good tool support (e.g. VS Code) - there is a Json schema for Json schemas which helps a lot.
The main advantage of the approach in the article appears to be that its easy to extend validation with code - which is a nice touch. Pretty much any complex Json validation I have written has to combine both schema based validation and code based checks - usually done completely separately.
Json-schema itself scales down pretty well. But the libraries for validating it do not. Even if your schema is only {}, the build size increment because of pulling in a jsonschema validator is enormous because of all these supported features.
> Json schema does scale down pretty well (indeed as the link above shows the simplest possible Json schema is just "{}")
I've not used JSON Schema so far, but I think showing the simplest possible definition does not tell much about how complex or easy to use a language is.
E.g., it's easy to write C code for a working program that does nothing or just prints "Hello World". The resulting code is very simple and can be written without much experience in C.
However that doesn't make C a simple language - and indeed writing programs in C that perform useful tasks is a lot more complex and requires more knowledge about the language.
In the same way, it's nice that Json Schema has a compact syntax for the "do nothing" schema, but I think a comparison between practically used examples would be better.
(Also the "do nothing" schema would have to compare against not doing schema validation at all - and compared to that, even the two brackets are still pretty costly)
I wrote a minimal validator in python that supports a subset of JSON Schema. I needed just a few basic validation features for my project, and didn't want to pull in a full implementation as a dependency.
can you tell me a bit more about this? i wanted something like this for a small api i have but ended up using marshmallow to validate jsons against defined schemas which i think is a bit overkill.
In Swift optional types are marked by a question mark (exactly as I suggested). Unwrapping an optional makes it non-optional, so the exclamation point has the opposite meaning.
Karate[1] has its own dsl for describing both json and xml. The output is short, but not very readable. For people new to the project it's rather cryptic. It is easy to maintain though once your team becomes fluent in Karate.
This has already been said in the nested comments, but why not “?” for optional? Coming from Typescript, Swift, etc “!” feels more like an assertion of non-nullity (is that a word?). It’s also easy to read as “not <key>”.
This is OK for simple structures, but in a deeply nested structures and larger APIs you'd probably also need some equivalent of json-schema $refs and ability to reuse blocks in your api schema.
Maybe I'm in the minority but I don't want light-weight at the expense of compatible/standard. It's being recognized that validation is more important than we'd expected - the recent ACM guidelines for moderation on Postel's Law really resonated with my experiences with RESTful/JSON micro-services (https://queue.acm.org/detail.cfm?id=1999945). But if we're going to the effort of validating, let's do it in a language/framework agnostic way ... jsonschema.org is mentioned in other comments but I'm a fan of the OpenAPI initiative (https://www.openapis.org/). Then we can all share the validation rules!
We're counting that against products when we're analyzing what services to adopt/purchase. You doing nothing at all ... or something non-standard results in everyone else having to attempt to do it for you with the resulting inconsistencies. I'm not sure why you think OpenAPI is tailored for XML ... Here's the specification for the sample application (petstore) with service and model definitions in YAML - https://github.com/OAI/OpenAPI-Specification/blob/master/exa....
I think we are talking about different things here. I love OpenAPI and wouldn't do a project without it. verify-json is a lightweight way to validate JSON schema and does not intend to change/improve anything about OpenAPI.
This is more for client-server JSON schema validations, comes handy when different teams are writing client and server side code. Especially in startups where iterations are rapid.
A long time ago in a galaxy far, far away, Corba implemented ICDs as "contracts" between parties. It seems like a client and server developed by different teams is still a good use-case for "something" that specifies the contract. In the 2000's, we wrote mil-spec style ICD documents that we attempted to enforce when writing code that had to interact (via a message bus). The benefit of a language/framework agnostic way of performing with something executed in code is a great step forward from just having an ICD document. Note that early Corba, XML-RPC and SOAP/XML could perform this function but people did horrible things to the data (using an indexed array of strings as parameters) that broke both typing and the ability to do type-specific validation.
From the way you've phrased your comment, I'm going to assume that the different teams are within the same start-up ... having a versioned, executable is even better when iterations are rapid if there are breaking changes. I've been in enough start-ups to know that things are skipped over to get a product out the door. I also know that the marketing and sales guys will say "we're already using a RESTful/JSON API for our back-end, let's open our API to third-parties to accelerate adoption of our SaaS". Ouch! (but I've been there)
I will admit that "back-filling" an OpenAPI specification for an existing service can be pretty daunting ... it's quite a bit easier if you start with the first MVP and iterate the ICD with each iteration of the client and server. Admittedly, we (software developers) still have issues versioning services. It's not so bad for incremental changes but breaking changes to a service where you have to support both older and newer clients is painful.
You can also use fefe (https://github.com/paperhive/fefe/) to validate any data with its pure functional and minimalist approach. Bonus: it's 100% type-safe automatically if you use Typescript.
I read the post. It seems to be mainly applicable to statically-typed languages. Apparently, the core problem with validation is "shotgun parsing":
> Shotgun parsing is a programming antipattern whereby parsing and input-validating code is mixed with and spread across processing code—throwing a cloud of checks at the input [...]
> The problem is that validation-based approaches make it extremely difficult or impossible to determine if everything was actually validated up front [...]
Err, what? So validating against a well-defined schema won't necessarily cause this. But okay, again, I can buy the benefits for statically typed languages. There's more though:
> Don’t be afraid to parse data in multiple passes. Avoiding shotgun parsing just means you shouldn’t act on the input data before it’s fully parsed [...]
> Use abstract datatypes to make validators “look like” parsers. Sometimes, making an illegal state truly unrepresentable is just plain impractical given the tools Haskell provides, such as ensuring an integer is in a particular range.
I've experienced this in Java/Jackson (which btw, proves this is not exactly new, sexy, or rare in the statically typed world).
What is the suggestion for a dynamic language? In e.g. Javascript, classes will only get you so far, and seems needlessly heavyweight if you aren't going to get other benefits of type-safety. I really don't see how this is helpful, even after putting in the time to investigate this.
In JS, my preferred way of handling json is heavily inspired by the elm json decoding way.
I want my decoders to essentially do 2 things:
1. Transform received data into a data structure that is best suited for my app. This might involve converting lists to objects, objects to sets, parse dates etc.
2. Only succeed on valid data
This way, my application never has to deal with bad data. Also, I get to design the data structures I use, not the APIs I use. By only validating and not transforming, you are pushing more advanced validation further away from where the data was received (since you'll need to transform the data at some point anyway).
76 comments
[ 3.2 ms ] story [ 147 ms ] threadProjects like this, mypy and others makes the whole thing bizarre to say the least.
- I was afraid it would change the language culture
- It was not comfortable to use
- The idea and design was, as you said, bizarre
- Use use a dynamic language, why the hell would I want that?
Years after the fact, I'm really happy about it.
First, most of my Python code don't use them, so Python stays python. But if I ever need them, there are here.
Of course, you could argue I should just use a language that has been designed for that since I need type. But that's missing the point: I don't usually start using hints. I code regular Python, because it's awesome. I just sometimes add type hints after the fact as a bonus.
The ability to do that is fantastic, even if the type system is far from perfect.
I want to code with Python most of the time anyway. I personally have very few use cases for another language, doing mostly web stuff.
Type hints are still very verbose, and convoluted, although it will get better with 3.9 thanks to 2 PEP targeting type hints ergonomics. So sure, it's never going to fit perfectly, but I'm glad it's here.
Now for a JSON schema, I guess it's the same deal. You use JSON because it fits your use case, and one day you realize your situation could now benefit from more robustness, and you add it to the mix.
Again, you could argue you should have though of that at the beginning, but projects don't have frozen requirements, they evolve. And often, you want to start lean, because it will most probably stays that way, and otherwise would be so costly the project would never grow to a point you would need robustness.
Now in this particular case, this lib is not just checking type, but also arbitrary logic. And making sure data is consistent is necessary no matter how dynamic or static your language is. There is a limit to the constraints your type system can represent.
Type inference can get you so so far
But then again, Django as well.
What's different this and Python type hints is that this was a public-facing API, so you have far less control over what data you'll see, and the server code was Java, so getting things strongly typed and validated early on makes your life easier.
My point is, I'd rather have a not so perfect one when I need it, that None at all.
And most of the time I don't need one.
I mean the attitude that Python is "good enough" because it has type hinting and therefore it is not worth bothering with exploring other languages, especially ones that have wholly more powerful type systems. I have found (personal experience only here) that it can be tremendously hard to get Python programmers to learn and embrace other languages and approaches to software.
If your program produces or consumes JSON _and_ you think it would be a good idea to ensure said JSON is what you are/a client is expecting, then you need something like this in any implementation language.
what I completely don't understand is the current fad on everything-should-be-yaml where yaml is marginally easier to write than XML, a tiny bit more than marginally easier to read (caveat emptor - Norway has a country code) and otherwise exactly as broken. (i'm talking about the 'safe' variant.)
I think that would be a great feature but from what I see static typing fails here, too.
NB Last time I used Pascal was probably 35 years ago....
Edit: I would suspect Ada would have something like this.
Pretty much all of them. Any simple predicate like this can be encoded with witness types.
Here's an example in Java, which is hardly the paragon of static typing (i.e. it's no Haskell/Idris/Agda/Rust/Typescript):
Now code can freely use `AlphaNumericString` and be guaranteed that it has been validated.You may object and say that newtype wrapping is cumbersome but:
1. That's an argument about sugar and ergonomics, not about the semantics that the static type system enforces
2. Some languages make it easier to generate forwarding methods to the underlying type (a la https://kotlinlang.org/docs/reference/delegation.html)
3. The `AlphaNumericString` is describing a smaller set of values than `String`. In general, you should be strongly considering the methods you allow and make sure that all paths continue to enforce the semantics you intend.
What led you to choose “!” as the “optional” modifier? My intuition would have guessed that character to have the opposite meaning.
[1]: project: https://github.com/tylerchr/jstn/ — playground: https://tylerchr.github.io/jstn/
Simple stuff like:
The basic idea was to use a file full of obiwan types to document the rest api. The schema was both human-readable and valid python. The api could then trivially check the schema. Once the schema is checked, of course, its safe to go walk the json knowing that things aren't going to crash on you etc.It worked great. Still works great. I still prefer the syntax to that of MyPy.
No real adoption, of course :) But fun to remember what could have been :)
lat = custom validator
b = shorthand for boolean (as is s for string)
! = optional
So for me reading the whole example its very easy to understand and digest.
and now i see lodash is a dependency.
It would seem much more natural to make them JSON structures since they're almost that anyway.
The main advantage of the approach in the article appears to be that its easy to extend validation with code - which is a nice touch. Pretty much any complex Json validation I have written has to combine both schema based validation and code based checks - usually done completely separately.
I've not used JSON Schema so far, but I think showing the simplest possible definition does not tell much about how complex or easy to use a language is.
E.g., it's easy to write C code for a working program that does nothing or just prints "Hello World". The resulting code is very simple and can be written without much experience in C.
However that doesn't make C a simple language - and indeed writing programs in C that perform useful tasks is a lot more complex and requires more knowledge about the language.
In the same way, it's nice that Json Schema has a compact syntax for the "do nothing" schema, but I think a comparison between practically used examples would be better.
(Also the "do nothing" schema would have to compare against not doing schema validation at all - and compared to that, even the two brackets are still pretty costly)
I like how compact it turned out: https://github.com/healthchecks/healthchecks/blob/master/hc/...
Usage examples in tests: https://github.com/healthchecks/healthchecks/blob/master/hc/...
[1] https://github.com/intuit/karate
One of the benefit of JSON over XML is that it is concise and fast to work with. The standard should reflect that part as well.
With this lib, as a developer to verify a JSON for a simple REST request is as simple as - `verify(json, "{a,b,c}")`
We're counting that against products when we're analyzing what services to adopt/purchase. You doing nothing at all ... or something non-standard results in everyone else having to attempt to do it for you with the resulting inconsistencies. I'm not sure why you think OpenAPI is tailored for XML ... Here's the specification for the sample application (petstore) with service and model definitions in YAML - https://github.com/OAI/OpenAPI-Specification/blob/master/exa....
This is more for client-server JSON schema validations, comes handy when different teams are writing client and server side code. Especially in startups where iterations are rapid.
From the way you've phrased your comment, I'm going to assume that the different teams are within the same start-up ... having a versioned, executable is even better when iterations are rapid if there are breaking changes. I've been in enough start-ups to know that things are skipped over to get a product out the door. I also know that the marketing and sales guys will say "we're already using a RESTful/JSON API for our back-end, let's open our API to third-parties to accelerate adoption of our SaaS". Ouch! (but I've been there)
I will admit that "back-filling" an OpenAPI specification for an existing service can be pretty daunting ... it's quite a bit easier if you start with the first MVP and iterate the ICD with each iteration of the client and server. Admittedly, we (software developers) still have issues versioning services. It's not so bad for incremental changes but breaking changes to a service where you have to support both older and newer clients is painful.
https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
I really like Elms way of Json decoding for this.
> Shotgun parsing is a programming antipattern whereby parsing and input-validating code is mixed with and spread across processing code—throwing a cloud of checks at the input [...]
> The problem is that validation-based approaches make it extremely difficult or impossible to determine if everything was actually validated up front [...]
Err, what? So validating against a well-defined schema won't necessarily cause this. But okay, again, I can buy the benefits for statically typed languages. There's more though:
> Don’t be afraid to parse data in multiple passes. Avoiding shotgun parsing just means you shouldn’t act on the input data before it’s fully parsed [...]
> Use abstract datatypes to make validators “look like” parsers. Sometimes, making an illegal state truly unrepresentable is just plain impractical given the tools Haskell provides, such as ensuring an integer is in a particular range.
I've experienced this in Java/Jackson (which btw, proves this is not exactly new, sexy, or rare in the statically typed world).
What is the suggestion for a dynamic language? In e.g. Javascript, classes will only get you so far, and seems needlessly heavyweight if you aren't going to get other benefits of type-safety. I really don't see how this is helpful, even after putting in the time to investigate this.
I want my decoders to essentially do 2 things:
1. Transform received data into a data structure that is best suited for my app. This might involve converting lists to objects, objects to sets, parse dates etc.
2. Only succeed on valid data
This way, my application never has to deal with bad data. Also, I get to design the data structures I use, not the APIs I use. By only validating and not transforming, you are pushing more advanced validation further away from where the data was received (since you'll need to transform the data at some point anyway).
As for libraries, I want composable parsers. For TS I'd probably use: https://github.com/paperhive/fefe/