I really wish lisps were more popular (or, really, popular again). Most people can't make it past the non-Algol syntax, which is silly IMO. But they do also demand more of the user than a typical language.
Their use of metaprogramming doesn't just allow you to extend the language, it really expects that of the programmer. Which means you have to assume the role of language designer to some extent. Learning how to do that definitely feels like a way to level up your skills. But it seems uncommon for people to want to do that.
I was really interested in lisps for a couple of years, but eventually I came to the conclusions: it's just hard to read. I know they say "the parens disappear" but even if that is the case, it simply requires you to jump around the expression holding lots of context in your head. I'm a fan of code that mostly reads top-to-bottom, left-to-right.
> But it seems uncommon for people to want to do that.
It becomes more obvious once you start managing developers vs being a solo dev. everyone making their own designer means the language can morph into a completely insular creation with a learning curve that expands exponentially with every new hire. A little extra boilerplate is the cost of standardized idioms that work across both your codebase that your new hires are already familiar with from working in that language at other companies. its why go was created. personally I prefer rust and elixir as good middle grounds.
This is a common reaction/belief but usually from people who have not actually managed a team of Lisp devs.
Lisp devs are managed in the same way as any other: You have style guidelines, design review, code review, etc. Sometimes a new macro is good and vastly simplifies code. It's accepted as a PR and documented like anything else. Sometimes a new macro is bad, and it's promptly rejected by the team. It's a persistent myth that Lisp programmers are just going to design their own little languages everywhere in a shared code base and it'll be impossible to understand.
(Case in point: Look at open source Lisp code. There isn't even management or code review there! Yet the vast majority of Lisp code is actually just functions and classes, with the occasional macro to reduce boilerplate. In some circumstances, you have a library offering a macro, and it's actually well documented and easy to understand. See Iterate, SERIES, etc. for actual examples.)
Rust or Elixir or Java or whatever aren't at all immune to monstrosities created by astronomically complex or baroque abstractions, FactoryFactoryFactories, and so on. How do teams avoid those? Style guidelines, design review, code review, etc.
It's hard to reconcile that with the comment they were responding to that claimed that lisp requires you to be the designer of the language a bit. I don't know enough to know who is right, but if the majority of the code is "just" regular functions and classes then I'd argue that it doesn't require custom design as a much as allow it, and the solution you're proposing is to mostly disallow it by convention. Like the comment you're responding to suggests, it's hard for me to imagine why having that much flexibility is worthwhile if it's going to be mostly unused when you can still write that occasional macro you call out in Rust or Elixir.
The first rule of writing Lisp macros is to not do it if you can get away with using functions. I'd still argue that it requires custom design, but in the same way OOP does. There are established patterns that you know and sometimes iterate on.
It’s going to be used. But at a project and team level rather than at an individual level. You go from lisp, then end up with a DSL that fits the project.
I think people underestimate how pragmatic meta programming can be because there are some obvious downsides. Arguably one of things that made Rust so popular was its inclusion of procedural macros.
But beyond that the thing I don't understand about the modern hate towards macros is that they are simply very fun.
Yeah I strongly agree. I think the issue is that metaprogramming is complicated, so people (especially early in their careers) tend not to do it themselves, and only notice it when it's making their lives difficult. But there are a lot of cases where a little bit of judicious metaprogramming makes life MUCH easier. If you treat metaprogramming as a first-class tool, countless rough edges will smooth themselves out for you—everything from `#derive[Clone]` to `#derive[Serialize]`.
As someone who is "into" programming languages (and making toy implementations of them), I think some of the most important macros are along the lines of Rust/Haskells `derive/deriving` for quickly enabling serialization, printing etc. Using a language without such capability quickly becomes frustrating once you move to any kind of "real" task.
In any kind of real task, serialization is not the hard part.
If you can write a meta program for it, you can execute that in CI and spit out generated code and be done with it. This is a viable approach in any programming language that can print strings to files.
It’s not frustrating, but maybe it feels tacky. But then you shrug and move on to the real task at hand.
You say that, but I've run into real production problems which were ultimately caused by bad serialization tooling. Language semantics are never going to be your biggest problem, but rough edges add up and do ultimately contribute to larger issues.
Dealing with nested association lists is a pain. this let you write your code with a dot notation like jq.
Macros are not only for solving a particular task (serialization, dependency injection, snippets,…) they let you write things the way it makes sense. Like having html-flavored lisps for template, sql-flavored lisp for query,… Lisp code is a tree, and most languages are trees, so you can bring easily their semantic in lisp.
Ruby, Python, and Typescript use metaprogramming rather heavily. They lack the homoiconic property of lisps, but they can do both higher-order functions and monkey-patching.
Meta-heavy code usually offers a nice DSL, but is proportionally harder to drill down through.
Lisp code is a tree, which fits how most languages are written. So it’s easy to embed other languages in lisp. But other languages grammars are very cumbersome and can’t fit one another.
Three thoughts (in the context of Common Lisp specifically):
- Every day that passes, the gulf between Lisp's tooling and what a typical user expects grows wider. It needs to escape Emacs and SLIME to something that feels complete and polished.
- There needs to be a little bit of a culture shift around Lisp to actually write programs that do things. How many programs can you download via apt or brew that are written in Lisp? They're executables at the end of the day so nothing in principle stops this from happening, but there's just a thread of modern Lisp culture where it's more fun to play around in the REPL and write creative libraries than to ship. (There are notable exceptions of course.)
- I personally like the quirkiness of Common Lisp, but there are so many ways to write it (imperative, functional, etc.), so many ways to structure your programs (one package, package per file, package inferred system, etc.), and so many ways to offer APIs (plain old data and functions, generic function protocols, etc.) that it makes it a combination of confusing and intimidating. I think shifting toward something a little more structured and disciplined like Coalton, while still giving the escape hatches to all of Common Lisp, would help a lot of people "join in" on building new code or building upon existing code.
> - Every day that passes, the gulf between Lisp's tooling and what a typical user expects grows wider. It needs to escape Emacs and SLIME to something that feels complete and polished.
Can you give specific examples of "what a typical user expects" that are missing from Emacs-based programming environments (SLIME, and/or others)? I'm not suggesting there aren't any, I'd just like to know your list.
Lisp does not have that much syntax for highlighting to be a problem.
Lisp is also a symbolic language. Meaning the code work on symbols, not data, only at evaluation the value of the symbol is known. There’s a lot of symbols manipulation routines like macros, intern, package loading,… that prevent to statically know the code.
Agreed. I think Clojure strikes a pretty reasonable balance here. It's opinionated about the programming paradigm, scales back some of the pain that comes from reader macros, and solves some of the bootstrapping problems by compatibility with other JVM languages.
I love clojure but the points still stand, kind-of.
- There is Calva for VS Code but the community default is emacs and cider
- How many programs in apt or brew are written in clojure? I'd concede that the community is great and focused on productivity, but it's so niche that you don't see much work out there made in clojure, and there is also a vestigial lisp sentiment to prefer building your own library from scratch instead of contributing to a standard library, which spreads the efforts of a small community too much
- Third one you need to mutate it a little bit: clojure is opinionated instead of having "so many ways", but its opinions, while great, are foreign to most programmers
> There is Calva for VS Code but the community default is emacs and cider
Emacs isn’t required. You can always create a REPL plugin. Emacs just does a lot of heavy lifting for you due to comint, sexp navigation, and process management being included.
> building your own library from scratch instead of contributing to a standard library
Simple data structures lead to very generic function. You don’t have to write tower or massive spread of abstractions like in Java or TypeScript. A struct is nothing than a hashmap that can help a typechecker. Most lisp programs prefer primitives or functions instead of manipulating complex objects -never ‘buffer.name’ but ‘(get-buffer-name buffer)’-.
From a module, what you need are functions and an opaque state holder.
With such philosophy, you don’t need a lot of libraries, which are often designed to be complex, when you need a simple model.
> Third one you need to mutate it a little bit
You don’t. Clojure already does the optimization for you for the standard data structures, and they are the only things you need in most cases.
the "assume the role of language designer" part is underrated. even if you never ship production lisp, writing a toy interpreter changes how you think about every other language you use. suddenly you understand why certain design decisions were made, not just what they are.
i wrote a tiny scheme evaluator as a learning exercise and it was probably the best investment of a few weekends i've ever made as a programmer.
Another reason this is interesting beyond "Lispiness" instead: implementing a (compliant) Scheme takes on some interesting problems that other languages punt - proper tail recursion, delimited continuations (off the top of my head, I don't know other methods apart from CPS), and hygienic macros.
I really wish lisps were more popular (or, really, popular again). Most people can't make it past the non-Algol syntax, which is silly IMO. But they do also demand more of the user than a typical language.
Their use of metaprogramming doesn't just allow you to extend the language, it really expects that of the programmer. Which means you have to assume the role of language designer to some extent. Learning how to do that definitely feels like a way to level up your skills. But it seems uncommon for people to want to do that.
I was really interested in lisps for a couple of years, but eventually I came to the conclusions: it's just hard to read. I know they say "the parens disappear" but even if that is the case, it simply requires you to jump around the expression holding lots of context in your head. I'm a fan of code that mostly reads top-to-bottom, left-to-right.
> I'm a fan of code that mostly reads top-to-bottom, left-to-right.
Programs are rarely linear; why do you expect code to be?
> But it seems uncommon for people to want to do that.
It becomes more obvious once you start managing developers vs being a solo dev. everyone making their own designer means the language can morph into a completely insular creation with a learning curve that expands exponentially with every new hire. A little extra boilerplate is the cost of standardized idioms that work across both your codebase that your new hires are already familiar with from working in that language at other companies. its why go was created. personally I prefer rust and elixir as good middle grounds.
This is a common reaction/belief but usually from people who have not actually managed a team of Lisp devs.
Lisp devs are managed in the same way as any other: You have style guidelines, design review, code review, etc. Sometimes a new macro is good and vastly simplifies code. It's accepted as a PR and documented like anything else. Sometimes a new macro is bad, and it's promptly rejected by the team. It's a persistent myth that Lisp programmers are just going to design their own little languages everywhere in a shared code base and it'll be impossible to understand.
(Case in point: Look at open source Lisp code. There isn't even management or code review there! Yet the vast majority of Lisp code is actually just functions and classes, with the occasional macro to reduce boilerplate. In some circumstances, you have a library offering a macro, and it's actually well documented and easy to understand. See Iterate, SERIES, etc. for actual examples.)
Rust or Elixir or Java or whatever aren't at all immune to monstrosities created by astronomically complex or baroque abstractions, FactoryFactoryFactories, and so on. How do teams avoid those? Style guidelines, design review, code review, etc.
It's hard to reconcile that with the comment they were responding to that claimed that lisp requires you to be the designer of the language a bit. I don't know enough to know who is right, but if the majority of the code is "just" regular functions and classes then I'd argue that it doesn't require custom design as a much as allow it, and the solution you're proposing is to mostly disallow it by convention. Like the comment you're responding to suggests, it's hard for me to imagine why having that much flexibility is worthwhile if it's going to be mostly unused when you can still write that occasional macro you call out in Rust or Elixir.
The first rule of writing Lisp macros is to not do it if you can get away with using functions. I'd still argue that it requires custom design, but in the same way OOP does. There are established patterns that you know and sometimes iterate on.
It’s going to be used. But at a project and team level rather than at an individual level. You go from lisp, then end up with a DSL that fits the project.
I think people underestimate how pragmatic meta programming can be because there are some obvious downsides. Arguably one of things that made Rust so popular was its inclusion of procedural macros.
But beyond that the thing I don't understand about the modern hate towards macros is that they are simply very fun.
Yeah I strongly agree. I think the issue is that metaprogramming is complicated, so people (especially early in their careers) tend not to do it themselves, and only notice it when it's making their lives difficult. But there are a lot of cases where a little bit of judicious metaprogramming makes life MUCH easier. If you treat metaprogramming as a first-class tool, countless rough edges will smooth themselves out for you—everything from `#derive[Clone]` to `#derive[Serialize]`.
As someone who is "into" programming languages (and making toy implementations of them), I think some of the most important macros are along the lines of Rust/Haskells `derive/deriving` for quickly enabling serialization, printing etc. Using a language without such capability quickly becomes frustrating once you move to any kind of "real" task.
In any kind of real task, serialization is not the hard part.
If you can write a meta program for it, you can execute that in CI and spit out generated code and be done with it. This is a viable approach in any programming language that can print strings to files.
It’s not frustrating, but maybe it feels tacky. But then you shrug and move on to the real task at hand.
You say that, but I've run into real production problems which were ultimately caused by bad serialization tooling. Language semantics are never going to be your biggest problem, but rough edges add up and do ultimately contribute to larger issues.
Lisp macros are more for not having to write the same type of code (all subtly different, but sharing the same general structure).
One such example is the let-alist macro in elisp
https://www.gnu.org/software/emacs/manual/html_node/elisp/As...
Dealing with nested association lists is a pain. this let you write your code with a dot notation like jq.
Macros are not only for solving a particular task (serialization, dependency injection, snippets,…) they let you write things the way it makes sense. Like having html-flavored lisps for template, sql-flavored lisp for query,… Lisp code is a tree, and most languages are trees, so you can bring easily their semantic in lisp.
Ruby, Python, and Typescript use metaprogramming rather heavily. They lack the homoiconic property of lisps, but they can do both higher-order functions and monkey-patching.
Meta-heavy code usually offers a nice DSL, but is proportionally harder to drill down through.
Lisp code is a tree, which fits how most languages are written. So it’s easy to embed other languages in lisp. But other languages grammars are very cumbersome and can’t fit one another.
And yet we have done it across most modern languages, with a lesser experience as Common Lisp or Scheme.
Lisp is versatile as all get-out, so you can program however you want. For example, we can roll like it's 1969:
I've been playing around with dropping most of the parens, but keeping the rest.
https://gitlab.com/codr7/shik
People have played around with those exact ideas for decades. No-one has ever come close to making it work.
Three thoughts (in the context of Common Lisp specifically):
- Every day that passes, the gulf between Lisp's tooling and what a typical user expects grows wider. It needs to escape Emacs and SLIME to something that feels complete and polished.
- There needs to be a little bit of a culture shift around Lisp to actually write programs that do things. How many programs can you download via apt or brew that are written in Lisp? They're executables at the end of the day so nothing in principle stops this from happening, but there's just a thread of modern Lisp culture where it's more fun to play around in the REPL and write creative libraries than to ship. (There are notable exceptions of course.)
- I personally like the quirkiness of Common Lisp, but there are so many ways to write it (imperative, functional, etc.), so many ways to structure your programs (one package, package per file, package inferred system, etc.), and so many ways to offer APIs (plain old data and functions, generic function protocols, etc.) that it makes it a combination of confusing and intimidating. I think shifting toward something a little more structured and disciplined like Coalton, while still giving the escape hatches to all of Common Lisp, would help a lot of people "join in" on building new code or building upon existing code.
New projects are making the newcomer experience easier:
- ICL https://github.com/atgreen/icl/ a full featured REPL in the terminal and the browser.
- JSCL's playground 100% in the browser https://wiki3-ai.github.io/jscl-kernel/ (very new)
- constantly new editor plugins. A new one: Zed https://github.com/etyurkin/zed-cl (all editors, for readers, including VSCode, Pulsar etc: https://lispcookbook.github.io/cl-cookbook/editor-support.ht... Those editors appeared in recent years. So, I see a good trend in the ecosystem).
> - Every day that passes, the gulf between Lisp's tooling and what a typical user expects grows wider. It needs to escape Emacs and SLIME to something that feels complete and polished.
Can you give specific examples of "what a typical user expects" that are missing from Emacs-based programming environments (SLIME, and/or others)? I'm not suggesting there aren't any, I'd just like to know your list.
Better syntax highlighting immediately comes to mind. Maybe code actions/automatic refactoring too.
Lisp does not have that much syntax for highlighting to be a problem.
Lisp is also a symbolic language. Meaning the code work on symbols, not data, only at evaluation the value of the symbol is known. There’s a lot of symbols manipulation routines like macros, intern, package loading,… that prevent to statically know the code.
It’s why people use the REPL flow.
Agreed. I think Clojure strikes a pretty reasonable balance here. It's opinionated about the programming paradigm, scales back some of the pain that comes from reader macros, and solves some of the bootstrapping problems by compatibility with other JVM languages.
I love clojure but the points still stand, kind-of.
> There is Calva for VS Code but the community default is emacs and cider
Emacs isn’t required. You can always create a REPL plugin. Emacs just does a lot of heavy lifting for you due to comint, sexp navigation, and process management being included.
> building your own library from scratch instead of contributing to a standard library
Simple data structures lead to very generic function. You don’t have to write tower or massive spread of abstractions like in Java or TypeScript. A struct is nothing than a hashmap that can help a typechecker. Most lisp programs prefer primitives or functions instead of manipulating complex objects -never ‘buffer.name’ but ‘(get-buffer-name buffer)’-.
From a module, what you need are functions and an opaque state holder.
With such philosophy, you don’t need a lot of libraries, which are often designed to be complex, when you need a simple model.
> Third one you need to mutate it a little bit
You don’t. Clojure already does the optimization for you for the standard data structures, and they are the only things you need in most cases.
Well CL is supposedly the programmable programming language. So none of this is surprising..
the "assume the role of language designer" part is underrated. even if you never ship production lisp, writing a toy interpreter changes how you think about every other language you use. suddenly you understand why certain design decisions were made, not just what they are.
i wrote a tiny scheme evaluator as a learning exercise and it was probably the best investment of a few weekends i've ever made as a programmer.
Another reason this is interesting beyond "Lispiness" instead: implementing a (compliant) Scheme takes on some interesting problems that other languages punt - proper tail recursion, delimited continuations (off the top of my head, I don't know other methods apart from CPS), and hygienic macros.
Last I checked no compliant Scheme required delimited continuations. Unless something like that was added in R7RS or later.