Talk:Metaprogramming

From Wikipedia, the free encyclopedia
Jump to: navigation, search
WikiProject Computer science (Rated Start-class, Mid-importance)
WikiProject icon This article is within the scope of WikiProject Computer science, a collaborative effort to improve the coverage of Computer science related articles on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.
Start-Class article Start  This article has been rated as Start-Class on the project's quality scale.
 Mid  This article has been rated as Mid-importance on the project's importance scale.
 

Metaprogramming… what non-functional-programmers call functional programming…[edit]

If you understand functional programming with functions as fist-class data-types, you know that there is no difference between code and data (or algorithms and data structures). So there is no point in a concept/term like “metaprogramming”. Everything is data. Everything is code. Then code can naturally modify itself. And data is just a language of a specific syntax that other code can process. It’s the generalization of all that, that is functional programming.

“Metaprogramming” really is the word that those use, who were not introduced to full-scale functional programming, or haven’t understood it. :)

94.220.250.151 (talk) 09:13, 23 November 2009 (UTC)

Not true, actually. The functional language Haskell has a specific language feature for metaprogramming: Template Haskell.--greenrd (talk) 03:47, 25 October 2010 (UTC)
I agree that functional programming is not meta-programming. The key difference is that the inside structure of a functional value can not be inspected by the program itself. Meta-programming requires that it is possible to observe and manipulate the inside of meta-programming fragments. So one can, for example, express logic for code like this: “if this function is (lambda x -> x+0) then change it to (lambda x -> x+2) but if it is (lambda x -> x-0) then change it to (lambda x -> x-2)”. Such a change, which depends on the code of the function, not the semantics, cannot be expressed with higher order functions in a functional language. --Krisrose (talk) 21:34, 27 June 2014 (UTC)

Metalanguage metaprogramming[edit]

The basic definition is not about functional programming. Nor is it a requirement to be able to modify itself while running.
"It means that a program could be designed to read, generate, analyse and/or transform other programs, and even modify itself while running."


A compiler would be an example of metaprogramming. It reads a programs source and generates an object program.


I would note that some metacompiler metalanguages are special purpose functional programming languages.
term = factor $(('*':MPY|'/':DIV) factor!2);
is a boolean function, calling other boolean (factor) functions, returning success or failure. Its input is the program source character stream. On success the input advances over characters matched transforming them into an abstract syntax tree on the pares stack. The term rule calls the factor rule that also creates an abstract syntax sub-tree. I.e.
MPY[factor,factor]     DIV[factor,factor]  or simply a  factor
        MPY                    DIV
      /    \                 /    \
  factor  factor         factor  factor
The term rule defines (recognizes and parses) a part of a programming language construct building an abstract syntax tree of the language construct recognized. This, tree building, side effect is a strictly defined part of the rule. making it more readable. "term" is part of an arithmetic. expression. The output sub-tree is pushed on the parse stack on a successful return. Above factor would be a sub-tree recognized by the factor rule. The :MPY or :DIV operations create the tree node and the ! operator builds the tree. The number following ! is the number of branches. !2 creates a two branch tree.
The syntax rules are metaprogramming. Not because they can change them selves while running. But because they operate on a source program transforming it into something else functionally equivelvent.

Steamerandy (talk) 18:44, 12 December 2014 (UTC)

Assembly & LISP[edit]

If you need to get examples of metaprogramming, read up on Assembly and LISP programming.. Especially about the use of pointers and how instructions and data are represented in a CPU.. You may realize something profound about historical programming that makes metaprogramming look relatively common..

New example[edit]

I just included the compiler metaprogramming example at the spur of the moment, probably someone else (or me) can do a much better job of it at a later time. --Anon

Subtleties of Metaprogramming- what is and not[edit]

Metaprogramming is quite a subtle concept. Sometimes it depends on the intention of the programmer rather than the form of the program. For instance, writing a program like:

DIM Radius AS Single
DIM Area AS Single
PRINT "Calculating the area of a circle"
INPUT "Enter radius of the circle: ", Radius
LET Area = 3.141592 * Radius * Radius
PRINT "The area of the circle is "; Area
END

is not an example of metaprogramming when the program is fed to an interpreter. However it is, when the program is fed to a compiler.

Note also that other programming methods can be thought of as metaprogramming. For instance, the genetic programming method or the automatic generation of a DB front end viewer/editor utility from the data model. --Derek Ross

I don't understand how the above program can ever be a metaprogram. My understanding is that a metaprogram is a program that manipulates or reasons about other programs. The program above does not do that. The compiler or interpreter, on the otehr hand are metaprograms.
Is dll use Metaprogramming? is threading Metaprogramming? --Tim32 09:28, 1 November 2007 (UTC)

Efficiency[edit]

I think I take issue with the statement that "This allows programmers to produce a larger amount of code and get more done in the same amount of time as they would take to write all the code manually." This falls into the same class as many other claims that ignore the fact that programming, done correctly, accumulates functionality exponentially, not linearly, with the amount of code written.

I'm think of claims that writing in (FORTRAN, C, 4GL, whatever) multiplies a programmer's productivity because "each line of FORTRAN compiles into ten lines of assembly language" or whatever (although, for some reason, the factor claimed is almost invariably ten). Yes, and once you've written a subroutine, the one-line subroutine call causes ten lines of code to be executed.

It seems to me that the relative advantages of writing subroutines (macros)(reusable objects) (whatever); versus inventing higher language levels (FORTRAN, Smalltalk, VB, whatever); versus various forms of code generation is very subtle and should not be thought of as a lines-of-code multiplier, but on the basis of whether the improved programming environment is a better fit to a) the programmer's cognitive model, and b) the problem domain.

I'd add that systems in which visible lines of code are produced are especially subject to marketing hype; I wish I could remember the ads a few years back that showed a page of greenbar paper with most of the lines highlighted in yellow, and the explanation "the yellow lines are the lines of code that XYZ wrote for you."

In most cases, it seems to me that the effect of most of these techniques is to give you a huge but constant additive head start, and a very small—probably too small to measure—productivity multiplier. --Dpbsmith 18:15, 8 Jan 2004 (UTC)

It is large enough to measure. I was taught that, for any given programmer/language combination, the time taken to solve the problems with a piece of code varied as the square of its size in characters for the following reasons: The chance of the programmer making a mistake varied approximately linearly with the number of characters in the code and the difficulty of finding the problem varied approximately linearly with the number of characters in the code. If that is true, then there is an obvious advantage in anything that drastically reduces the number of characters that the programmer needs to write in order to specify a program. The effect becomes more noticable even for a small reduction in size as larger programs are considered.
Certainly I noticed a large productivity increase when I moved from writing database reports in VB to using a "home-made" database report generator which writes those same types of report in VB for me. Time required per report dropped from three days to three hours. The main reason is that I am far less likely to accidentally include bugs due to typos because I am now "metaprogramming" a relatively short report specification rather than "programming" a fairly l arge VB program. --Derek Ross


Dpbmith is right. You were tought wrong. The number lines of code is not as big a factor as taught in school. A good programmer should have a good idea of were in a program the problem is from the bug symptoms. Modular coding development reduces the amount of code being debugged to small parts. Planning and having well defined modules allowed large projects to be worked on by many programmers. More code does equate to more possible typing errors. But most typos will be detected during compilation. I have programmed more in assemble then high level languages and at times I have spent more time getting a high level language to do what I wonted. I once spent days trying to get C++ to staticly load a constant structure that would have taken minuets to code in assembly. And in fact there would have been little difference in the number lines of code.


Programming and coding are not the same. Good programming comes from well thought out algorithms. Coding the algorithms using good coding practices is what determine the ease of coding, debugging, and maintainability of the code.


A FORTRAN programmer were I worked was claiming how much easer it was programming in FORTRAN then assembly. He had a simple report program assigned to him. I bet a lunch I could write it quicker in assembly and won. Had it working in less then 2 hours. Took him over two days. My assembly had many more lines of code. Ten to twenty times more lines of of code. The assembler caught all my typos. And there were quite a few. But it worked the first time it ran. The FORTRAN programmer was still working on getting formating right. A great example of using the wrong language for the task. COBOL would of been a better choice for writing a report.


On the other hand I have spent months because of the misunderstanding of assembly interrupt enable and disable instruction. Under my direction another programmer had coded the task switcher of an operating system we were developing. The interrupt enable and disable instructions were reversed. Looked right to me as other computers I had worked on used similar op code mnemonics. But on this machine the interrupt flag had the opposite effect. CLI enabled interrupts. Interesting thing is it only crashed once every three months or so. Holding off interrupts till the start of a critical code sequence made them more likely to occurs before the first critical code instruction reducing the chance of one during the short critical code sequence. Now this is exactly the opposite of the claimed program bugs being proportional to amount of code. Critical code (interrupt sensitive) is the shortest code sequences possible to reduce the time interrupts are disabled. And here ever one was bugged and very hard to find with it causing a problem once every three months. Most were around critical link changes less then 10 assemble instructions. I kept overlooking the CLI thinking it was disabling interrupts. To be fair one had to look at the hardware manual to understand CLI. The assembly manual said it cleared the interrupt enable flag. The hardware manual explained interrupts to be enabled when the interrupt enable flag is 0. SEI disabled interrupts.


Using a programming language suited to the problem reduces the cost of development and maintainability. Domain specific languages make since if designed right. However if a language is not easy to read and understand it's not going to be easy to maintain. Write once languages are good for a one time use application. A text editors like teco is a good example of a write once language that should not be used for general programming.
Self modifying code has been fround upon since the beginning of programming. It is hard to follow program logic of such programs.
Steamerandy (talk)

Examples: Suggestion[edit]

Might be interesting to: show the example programme in

  • Java
  • C or C++
  • other langauges

Might be interesting to: show a short example programme that does "something useful".


An addition[edit]

I have a good example of the metaprogramming using Lisp macros: [1]

It is possible to extend an article with this sort of examples (and it correlates with the idea of the compiler metaprogramming).


Metaprogramming: proposed definition[edit]

Here is a proposal definition.

Metaprogramming is the writing of a program by a programmer whose intended purpose is to generate or output code, for the purpose of alleviating the need to write the code that would be generated by the program.

This excludes cases such as

compilers: not programming by a programmer, code's intended output is not code
compiler-compilers: not a program, output is not code

Thoughts? --Dysprosia 09:34, 11 Mar 2005 (UTC)

I think it's much too specific, really. Metaprogramming isn't at all limited to code generation. In some cases it extends the capabilities of the language — for example, you can perform tasks using reflection, a form of metaprogramming, that you could not perform without it, including many that don't involve code generation at all. I struggle to come up with a good definition, because it's a sort of moving target — yesterday's metaprogramming is often no longer considered metaprogramming today. The best thing I can come up with is to say it's a term used to describe many, but not all, programming technologies that operate on programs. --Deco 10:37, 11 Mar 2005 (UTC)
Can I get an example of using reflection? Perhaps the definition above can still be salvaged. Note we do need a tight definition of what metaprogramming is or it allows the whole term to degenerate, if one says that using a compiler is metaprogramming, it effectively means that all programming is metaprogramming, &/c. --Dysprosia 10:56, 11 Mar 2005 (UTC)
Hmmm, your definition does not exclude compilers or compiler compilers yet, I think:
  • Compilers output code (that's a definition of compiler). Compilers are written by programmers.
  • Compiler compilers are programs that output compilers, which are programs.
  • +Comment: "By a programmer" is redundant, since all programming is done by programmers. It can even be confusing, since the point of metaprogramming is that part of the coding work gets automated, and thus taken over by the machine.
(some text on history of metaprogramming removed here)
Note that I disagree that the current definition says "All programming is metaprogramming", so I don't see why it needs changing. But I'm willing to see what you can come up with. :-)
Thank you very much for actually looking and trying by the way. It shows that VfD sometimes still works. :-) --Kim Bruning 12:15, 11 Mar 2005 (UTC)
I have explained why the current definition does so, since it includes compilers.
Re: compilers: you miss "whose intended purpose is to generate or output code", so merely using a compiler would not be considered metaprogramming, but writing a compiler would be (is that necessarily desirable?)
Re compiler-compilers: The input to a compiler-compiler is a definition of the grammar, not a program that outputs code. --Dysprosia 00:16, 12 Mar 2005 (UTC)
But the objective is to manipulate code, and writing a compiler would indeed be considered metaprogramming, as it should be. :-) So does that mean we were actually in agreement the whole time? ;-) --Kim Bruning 13:06, 12 Mar 2005 (UTC)
I don't know... I think metaprogramming has to serve some self-referential purpose; that is, one metaprograms to avoid having to write the code one would have to anyway without it, so if a compiler writer uses metaprogramming techniques to write the compiler, then that would be metaprogramming, but if a compiler writer merely writes a compiler for the hell of it, that wouldn't. Metaprogramming, I think, has a lot to do with the intent and the object of the produced code. But this sort of aspect of metaprogramming is less clear. --Dysprosia 23:06, 13 Mar 2005 (UTC)
Let's run through an example: I have to write a lot of code, but this code has some kind of pattern to it. So to refactor, I write some separate code that creates these patterns, and I write a separate file to feed to this program which specifies which patterns I want specifically. What's this act called, according to you? --Kim Bruning 23:37, 13 Mar 2005 (UTC)
It's best you make the comparison directly without me having to guess what point you're trying to make ;) I'm guessing you're saying that this is what compilers do. But still, you haven't looked at the intent aspect. If I write a compiler to help me write a compiler, then I would say that's metaprogramming. But if I write a compiler to write a database program, then that would probably be metaprogramming too, since it's writing a program to generate code that helps me in the project I'm working on, but if i'm writing a compiler, it's a bit of a special case, since for the action to be helping what i'm doing, it would have to help write the compiler. Clearer? --Dysprosia 08:27, 14 Mar 2005 (UTC)
Guilty as charged! :-) What I'm trying to do is to try to grasp what part of writing a compiler makes it non-metaprogramming, according to you. I think it's important to understand your opinion, before I try to convince you of mine or not. :-) --Kim Bruning 09:11, 14 Mar 2005 (UTC)
Like I've said, the intent and purpose of the use of the metaprogramming technique, to me, would determine whether it would be metaprogramming or not. If one argues that compiler writing is metaprogramming, then, one would have to look at the intent. If the compiler was written to help write the compiler, that would be metaprogramming. Otherwise, it would not be. --Dysprosia 01:38, 15 Mar 2005 (UTC)

In a number of famous algorithms, one may find that a "metaprogramming" technique is indeed used. For instance, consider the Knuth-Morris-Pratt algorithm. You will find that for any search string, a state machine is necessarily constructed to be run over input to find matching strings. In database work, to find the result set of a query in a reasonable amount of time, a query execution plan is generated. I often have to wonder what separates these techniques from the techniques mentioned here and if there is a true difference, or if its concept renaming.

Walid Taha's work (Metaml/Meta Ocaml) is certainly a unique variety, as you could say his programming model allows for (hopefully easy) meta programming, meta-meta programming, meta-meta-meta programming, etc. ad infinitum. But many others don't seem much different than embedding a compiler of some sort into the language and using it. Root4(one) 05:59, 26 November 2006 (UTC)

I prefer the more direct definition of

Meta-programming is the writing of programs that manipulate other programs as if they were data.

In this view the programmer who writes a compiler is engaged in meta-programming because the code they write -- the compiler -- manipulates the abstract syntax tree (AST) of some program and translates it into another data structure representing the assembler instruction sequence (for example) of the target program. (Compiler users are not engaged in meta-programming because the program they write and compile does not itself manipulate code as data.) --Krisrose (talk)

I agree except when using a compiler to write a compiler. Most C compilers are written in C for example. This may be obvious to most but for the few who like a computer work on what has been stated the exception needs also to be stated.

Steamerandy (talk) 22:44, 12 December 2014 (UTC)

history of meta-programming[edit]

This is a good article, congratulations to all the authors!

One useful addition might be a history of meta-programming. I came here interested to learn when the idea first started to surface (the idea of human-oriented meta-programming that is: not the compiler itself). E.g., the C++ article states that templates were added some point after 1990 (and I'm sure C++ was not the first language to provide explicit support for meta-programming either). -- Jon Dowland 09:18, 8 September 2005 (UTC)

Lisp is a much earlier example (sixties). --MarSch 12:46, 12 December 2006 (UTC)

Forth[edit]

There ought to be some discussion of the Forth programming language which has native support for metaprogramming through CREATE and DOES> and other words. The functions used to build functions are generally made available to the user and it's straightforward to flag newly defined words to act as compile-time metaprograms rather than as words to be compiled. sigfpe Dec 5 2005

Actually sense the FORTH language is made up of words in it's dictionary. Defining new words is extending the language, making a new dialect of the language, would be (I think) metaprogramming. FORTH freaks think FORTH is a metacompiler just because it compiles it's self. (creating a new instance of it's self) Totally ignoring that compiling it's self is just a self-hosting compiler. No different then a C compiler written in C. And thus if accepted would mean every self-hosted compiler would be a metacompiler. But on the other hand programming in forth is extending the language. Creating a dialect (or derivatives) of FORTH. So in essence programming in FORTH is metaprogramming and forth would be a metaprogramming language. I would never have considered forth a metacompiler. I have gotten into a argument with the creator of the metacompiler topic because of the use of the FORTH metacompiler definition he used. I have written a very advanced metacompiler that extended CWIC adding a machine assembler defining language allowing the defining of an assembly instruction operand syntax and the expansion of instruction into bit fields. Instruction or generated into bit addressable memory sections. It was used to write a COBOL cross compiler running on a DEC-System-10 (36 bit word machine) producing code for TI-990 a (8 bit) byte addressable, 16 bit word minicomputer. So you can understand my bias as to the definition of metacompiler. Looking at FORTH in metaprogramming terms it is a specialized metalanguage compiler/interpreter that extends it's self with every new defining of a word to it's vocabulary it creates a new dialect of FORTH.--Steamerandy (talk) 18:02, 19 November 2014 (UTC)

Flare[edit]

You should check out this page: http://www.flare.org . It's about a programming language specificly for metaprogramming.

PHP and the like[edit]

It seems to me that PHP, ASP, JSP, HTML::Mason, and the like, usually referred to as "Web templating systems", are a form of metaprogramming. The programmer writes code in one language - PHP, VB, Java, Perl, and the like - that generates HTML, instead of writing the HTML directly. You may argue that HTML is not a programming language, just a markup language. Nevertheless, the process is largely the same. Also, these tools can also insert JavaScript into the HTML, and that is definitely a programming language.

Note that the file they create may start with the "high-level language" at the outermost level, and escape into the markup language, or vice-versa. These two approaches are essentially equivalent.

I'm going to think about adding something to the page that describes this common example.

I would tend to agree with this. Its quite often that one needs to incorporate some javascript generation in an asp.net application, and in fact if one does some research, the technique is fairly common... But in fact I think its a touch overused. Often you can find asp.net code that generates javascript that might be better generated as calls to specific javascript functions included in separate javascript files. It is much easier to define the functionality you need, place the functionality behind a few functions, test said functions, and then invoke or place calls where needed. Unless you already have a huge javascript library to use, javascript generation does not always make sense.
These days, any javascript generated is pretty low level and hidden behind few abstractions. But it is still generated. Root4(one) 05:24, 26 November 2006 (UTC)

An external link?[edit]

I would like to suggest this article as a relevant external link. But, well, I'm the author of it, so I fell like it would be more appropriate for someone else to actually add it (if you find it relevant) LotLE×talk 06:53, 29 June 2006 (UTC)

other than the word "meta" this seem totally unrelated to metaprogramming. --MarSch 11:33, 1 November 2007 (UTC)

Viruses[edit]

Shouldn't computer viruses be included in the article, as they manipulate executables to copy themselves?--Theodore Kloba (talk) 21:24, 28 April 2008 (UTC)

Metaprogramming in C++ with Qt[edit]

Can "Qt's meta object compiler (moc)" be added as an example of meta programming? It generates more C++ code based on the contents of a .h file. Vdboor (talk) 10:33, 17 September 2009 (UTC)

Compile/Runtime[edit]

...or that do part of the work at compile time that would otherwise be done at runtime.

It makes more sense the other way around.

--Sina (talk) 13:10, 24 October 2010 (UTC)

No, that is the right way round.--greenrd (talk) 03:44, 25 October 2010 (UTC)
Please explain because if anything, metaprogramming gives us more control at runtime. This is also what's implied by “programs that modify themselves.” Programs modify themselves at runtime.
Even in case of macros, one can write a program that generates other programs at compile-time, that's true. However, if the macro expansion is not done at compile-time, it certainly won't be done at runtime either. In that case, the work will be done by the programmer who has to hard-code the program that would otherwise be generated. Sina (talk) 01:52, 28 October 2010 (UTC)
I think the vital word here is or. This is one of the benefits of, or use cases for, metaprogramming - not the only one. An example of this would be generating some never-changing data, such as the first 10,000 prime numbers, at compile time, and storing them in an array. This means that you don't have to generate the numbers on program startup. And nor do you have to store the numbers in your source code, which takes up space and leaves open the possibility that a programmer might accidentally hit a key, modifying one of the numbers, and not notice their mistake! I am perfectly aware that metaprogramming can be done at runtime, but it doesn't have to be.--greenrd (talk) 08:25, 28 October 2010 (UTC)
Thanks for clarifying. Or indeed is what I had failed to see. Sina (talk) 03:14, 24 August 2011 (UTC)

There is a lack if defination of reflection or disagreement with the reflection wiki topic.

It simply states reflection is a programs ability to modify itself. However wiki reflection states it as the ability to modify itself at runtime.

These two topics should agree. Steamerandy (talk) 16:59, 23 December 2014 (UTC)

Metacompiler[edit]

Metacompiler have long been defined as taking a matalanguage program and translating it into a compiler.

metacompiler from the XDS 940 TREEMETA manual(1968):

A metacompiler, in the most general sense of the term, is a program that reads a metalanguage program as input and translates that program into a set of instructions. If the input program is a complete description of a formal language, the translation is a compiler for the language.

Metacompiler definition as stated in CWIC (Compiler for Writing and Implementing Compilers) documents(1969-1985):

A metacompiler is a program whose input is a description of the syntax of a programming language, plus a description of its semantics, i.e., or its equivalent in terms of a machine language. The language in which such a description is coded is called a metalanguage. The output of a metacompiler is a compiler for the described language. This compiler can then be used to process a program written in that language: i.e., to examine it for syntactic correctness and to translate it into a machine language or its equivalent.

Now in the metaprogramming topic we have:

The ability of a programming language to be its own metalanguage is called reflection or reflexivity.

Now that fits a metacompiler written in it's own metalanguage compiling itself. It seams a special case of reflection equivalent to self-hosting compiler. Am I right in explaining a metacompiler compiling itself as:

The process of a metacompiler, written in it's own metalanguage, compiling it's self is called reflection, a metaprogramming term that in this case is equivalent to self-hosting compiler.

I have been programming sense 1965 and written many commercial compilers. My computer lingo is a bit old school. Could someone here check out the metacompiler topic. I have redid it based on metaprogramming terminology (I hope).

I made an RFC because the original author defined metacompiler way to much like the FORTH definition of "Written in it's own language, compiling itself this way" making it a metacompiler. When historically some metacompilers have been written on other languages (assembly, LISP, CL/1 and PL/1) and of course did not compile themselves. When I last looked at FORTH it was not being called a metacompiler. --Steamerandy (talk) 22:29, 19 November 2014 (UTC)

metalanguage[edit]

The statement:

The language a metaprogram is written in is called a metalanguage.

Is inconsistent with the defination of metalanguage.

Metaprogramming is programming that treats programs as data. Writing a compiler is metaprogramming. Compilers may be written in most any programming language. A metalanguage is used to describe other languages or talk about them. A compiler may be written in a metalanguage. But for example metaprogramming may be done in c++. Would then, c++ be a metalanguage?

Steamerandy (talk) 02:41, 27 November 2014 (UTC)

Article Quality[edit]

...is quite poor. And it's consistent throughout the entire article, not just a few things here and there. Unfortunately, I don't know anything about the subject, so I'm just going to tag it and run. Deacon Vorbis (talk) 01:57, 22 January 2017 (UTC)

Treat programs as their data[edit]

The definition given in the introduction is very restrictive ("treating programs as data"). It implies that metaprogramming is possible only within homoiconic (therefore interpreted) programming languages. It then talks about C++ metaprogramming, which is purely static, creating an worrying level of inconsistency in the article. Vincent Lextrait (talk) 01:17, 30 December 2017 (UTC)