How fast is awk

Have you implemented something like this before? And assuming you have 2 cores available, how much can it increase performance? Here we get rid of one more hash access than Fletch did. But he could do that in his code also. Pingback: Destillat KW duetsch.

On the String. It seemed to shave about 10 seconds off the runtime on my computer, which left it still slower than awk. I thought it could be thread communication overhead, but packing multiple lines into larger packets did not help. The code is available here. Uglier and more wordy than I thought. MAWK is still affected by a few bugs that seem to have been around for years now, and nobody seems to be interested with fixing them.

I have just stumbled upon this one. It seems to me that the issue of processing speed is almost irrelevant for data formating tasks. All tools seem fast enough. For a typical new data set, one needs to do it once and then never do it again. A lot more time is spent on writing and testing the code to accomplish this task. Based on these presumptions, the AWK implementations still beat everything else.

I am an AWK fan and I use gawk. This is kind of silly. The purpose of awk, etc is for a quick and dirty. Mawk, like dash is broken in subtle ways. Always script to standard available utilities or pay the price.

Why on earth would you use Ruby for this? Assuming your data set is formatted as you specified, C code follows. Linear search might be an issue if nelem gets overlarge, but likely not even close to a bottleneck.

Pingback: MAWK does not support length array? I have been using mawk for nearly a year now for very large data sets. I had switched to C to process my particularly large data sets, because nawk was too slow. I switched to mawk because I have found it a little faster than C, and much faster to code and debug. The speed increase of mawk over nawk is incredible. I compared mawk to gawk and mawk is 4 times faster than gawk.

I tested on a file with about one million lines. Very well done Mike Brennan! Do you have a link to the large datasets that causes mawk to give incorrect output? I would like to see if mawk can be fixed.

There is something called SCC. It is several times faster than MAWK in my benchmarks. But it is alpha version and needs GCC Would like to see a benchmark with an actual C version. The changes in the script were done based on advice on the Awka site on how to modify scripts for better performance. My Daily Feeds. I ran the benchmark with Lisp and several of the other languages that you used.

Things shuffled about a bit but mawk is still the clear winner. For some perspective, the gawk 5. Congratulations to the Ubuntu team for finally bringing mawk into the new millennium! Can I still say it's new 20 years in? For years at work I ran into a strange issue in the build system where when new people started they would get mysterious segfaults.

Install gawk and the problem went away. I would have filed a bug but it was so deep inside configure with several extra layers of indirection that I didn't bother. That project has since been retired thank god. I personally still use it on a regular basis maybe once every couple of weeks to pull a field out of a log file, sum up a column in a CSV file, etc. If all of the above are truly inadequate to the task at hand, I'm probably better off writing a Python script anyway.

It went downhill from there. It might cause a problem when used occasionally in the command line. But it will consume substantial kernel CPU if done on a large scale. Yes, you could implement these things in Python fairly easily, but with two downsides: You can't just write the script in your shell, you need to open a text editor. Though this is mostly a Python problem, caused by its use of whitespace for code flow. Awk uses brackets and semicolons, so this isn't an issue. I would wager that most awk scripts are written from within a shell.

Most "real" languages require you to do a bunch of boilerplate such as looping over input, or explicitly do conversions to non-string values.

For "real" programming languages, it makes sense to require this kind of boiler plate -- but awk lets you elide all of it because its only purpose is to execute programs over records and fields.

For a quick-and-dirty language like awk it's a godsend to not need to have any boilerplate. Plus you can set FS to be a regxp if needed. I have half a mind to submit a proposal to POSIX to add a new option, but there's no such extension in any implementation of cut that I've seen.

Pre-existing practice isn't a hard requirement, especially for the upcoming revision, but I feel like the fact it doesn't exist constitutes proof that cut is a lost cause and should be left alone. Still, modern programmers don't seem to understand the meaning of the word "efficient" Cheers, Wol.

Posted May 20, UTC Wed by geert subscriber, [ Link ] For small amounts of data, the tool usually doesn't matter at all. A reasonable pipeline will scale to millions of lines of text very easily, because the per-process overhead just isn't that big compared to the actual work being done. Many many moons ago someone wise corrects me the correct way, I was doing like everyone else in wild this: cat somefile grep somepattern and the correction was..

Cheers, Wol. I was thinking of some other tools like wc which have different output when given a filename versus reading from stdin. And it echoed the essence of UNIX, do one thing and do it well. It's a damn good tool to know. Posted May 20, UTC Wed by scientes guest, [ Link ] sed is not a horrible idea, but whenever I use it I run into the fact that it cannot parse arbitrary regular languages because of the lack of non-greedy matching i.

This is certainly true when parsing something like a path name into its constituent components. True non-greedy matching is more powerful than that, of course, but eventually you may want to reach for a Real Parser TM. Posted May 21, UTC Thu by xtifr subscriber, [ Link ] What I hate is to see people flailing around with a bunch of overspecialized and slow tools like cut, paste, comm, grep, sed, and so on, to do--poorly--what a trivial amount of awk would do cleanly and well.

Posted May 23, UTC Sat by NYKevin subscriber, [ Link ] Well, personally, I like to think that I'm adhering to the Unix philosophy each binary does one thing, and does it well, whereas awk seems to want to do "reading and modifying text" well, whatever that's supposed to encompass , but this will quickly degenerate into a flamewar.

Posted May 26, UTC Tue by jezuch subscriber, [ Link ] A counter-point from me would be that I took an almost immediate dislike to awk because I felt that a full-blown imperative language is overkill in a context which asks for a more declarative approach But I generally favor declarative and functional over imperative wherever that's practical. Posted May 25, UTC Mon by anton subscriber, [ Link ] If all of the above are truly inadequate to the task at hand, I'm probably better off writing a Python script anyway.

The nice thing about awk is that when you find that your non-awk shell-scripting tools miss a feature, you don't need to rewrite the whole shell script in python. It neatly fills the gap between Bash and Python. Quite a lot faster than Python, it does more in one to five lines. Indeed regarding it being a proto-Perl : Perl was described at the time of its introduction as a derivative of awk more suitable for heavy lifting and complicated multi-component programming.

I never did learn perl, because in my own work I just didn't run into problems that a couple of lines of awk couldn't handle, or for which some other language was clearly called-for, as actual design and engineering was involved.

I still use awk one-liners daily. They fit particularly nicely into shell scripts, in-place, too. Posted May 20, UTC Wed by edeloget subscriber, [ Link ] Given the fact that awk is heavily used in many scripts in the embedded world mostly through busybox awk it's definitely not going to disapear any time soon.

It may disapear one day, but not before shell scripts died which means we would have access to a better kind of shell. Similarly I just realized how odd is writing HTML tags into this form when all other forms I use requires Markdown and I automatically almost started to type it when I realized where I am Awk is much, much more approachable. If I'm trying to explain to someone who won't be doing a lot of scripting how to munge text, flat files, awk's by far the best option if sed isn't easier.

They'll likely be able to extend what they've learned because the marginal costs are low, and if awk's out of steam they're likely to need some guidance for other reasons. There was a glorious period where Perl was more common than Bash, after the BSDs and commercial Unices adopted Perl, but before the dark years of "shell scripting" becoming synonymous with "Bash scripting". If you had reason to venture away from POSIX utilities for system management tasks, Perl was the obvious and perfectly reasonable choice--it was and remains the better AWK.

I feel much better about continuing to use awk. I do stumbled and use it every now and then. Awk and Python fall in roughly the same performance tier for that kind of code. Also: "I have since found large datasets where mawk is buggy and gives the wrong result. Afaict, mawk's maintenance seems to be a bit up in the air the original maintainer basically disappeared years ago and hasn't blessed any successor, so the Debian-patched version became the de-facto current version, since at least it staved off bitrot.

I'm personally a little more comfortable with something actively maintained like gawk, despite the speed differences. I usually use n awk because it's the default on OpenBSD, but have to admit gawk's artificial-filesystem-based networking support is pretty cool.

Ultimately, what are you implying? Am I wrong? It's likely to be good enough for many things, though certainly prototyping , and it's definitely convenient for quick scripts.

Actually, I think the general term for this is a "pipe". I've seen it called "generate and test [programming]" in Prolog books, but that's specific to a filtering pipe. This proved to be quite useful when working with multiple Unices who all had different awks. Still, the One True Awk still has my favorite opening line in its "b.

Makes lots of sense for stream processing as you said. On the other hand, both awk and sed quickly spiral out of control if you need to do anything nontrivial that spans newlines.

If the unit of input in this kind of stream processing system doesn't match the problem domain exactly, things get very difficult very quickly. Awk isn't so bad if you're clever about RS, but sed sucks. A tragic gap in the Plan 9 legacy has been structural regular expressions, which deal with these situations adroitly. You can handle multi-line patterns in awk. My problem was that the records really were purely newline-delimited, but I needed to process them using information from their context in the stream.

Fair enough. That's beyond the common cases awk addresses. At that point, I just switch to Lua. I forget if you're a Python or Ruby guy. Regular expressions are my favourite secret weapon; So many problems are made simple by regular expressions and so few people outside of IT know of them.

I was curious enough that I bought and read it just at the end of summer. It really is excellent. Highly, highly recommended. You can't go wrong with any programming books Brian Kernighan co-wrote, really. Concise, with a lot of depth that reveals itself on repeat reading. I recommend it quite highly, too. Great language, great programming book. They're included as documentation in some BSD installations, and should be easy to find otherwise.

The intros to lex and yacc are particularly good. I'm glad you saw this thread. It's always nice to find out somebody actually paid attention to and appreciated some advice you put out on the interwebs. Many people consider Perl to be the next evolution of awk, but I prefer to think of awk as just the essentials of Perl.

Use awk. Useless Useless 4, 18 18 silver badges 20 20 bronze badges. RetroCode: python is more "general purpose" than perl; the equivalent one-liner will probably be much longer. RetroCode Because the CPython implementation's command line tool doesn't support being used as a filter for newline-terminated records, because it is more verbose, which is undesirable for one-liners, and because all regex modules I have tried with Python are 10 to slower than Perl's, which is bad for large data.

Thanks for this nice overview of all these programs. It really sheds light in the darkness. Which grep and which awk are you refering to? It's not really fair to the other utils that grep is just searching and they are also replacing. Those are completely bogus numbers.

Talk about comparing apples and oranges - it's like saying you can only find a new car on web site A in 5 secs whereas you can find a car, negotiate a price, get a loan, and purchase the car on site B in 1 hour so therefore site A is faster than site B. The article you quoted is completely wrong in it's statements of relative execution speed between grep, sed, and awk and it also says awk Show 4 more comments. Sign up or log in Sign up using Google. Sign up using Facebook.

Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. The Overflow Blog. Does ES6 make JavaScript frameworks obsolete?