[kwlug-disc] Scripting in Linux and unit.d
John Van Ostrand
john at netdirect.ca
Tue Jan 10 12:46:08 EST 2012
You are going to need to know regular expressions and, from the sounds of it RE sub matches.
If there was a line like;
Jan 1, 2012 12:34:56: User colin logged in.
You could match that in perl using
if (/ User (.*) logged in/) {
# Brackets are sub-match, it puts the matched string into $1
$user = $1;
}
Of course that RE also matches
Jan 1, 2012 12:34:50: User colin could not be logged in.
Except $1 would be set to "colin could not be" so you may need to get a little more specific.
The RE / User ([^ ]*) logged in/ would work much better. The [^ ] means anything but a space, so it would only fail if user names could have spaces.
Generally speaking you can choose how much specificity you want in the RE. In the examples above I'm matching partial lines, ignoring the date and time and the tailing period. You'll find your own happy medium, more specific usually means slower processing but more accurate results. If you know the full limit of all the text that appears in the logs and are confident it won't change you can be more general.
----- Original Message -----
> Thanks john i was thinking of some kind of ugly grep thing that would
> pull out the lines that start with
> [SEVERE] and dump them into a file worst case but this could certainly
> do me a solid favor. I would like to be able to seperate the server
> stuff from the user related things too but that might be a bit more
> complex filtration.
> The directory structure goes something like this
> /ServerRoot/Server.log
> Than
> /ServerRoot/Plugins/PluginConfigurationDir/pluginspecifics.yml
> If i could split the server.log to dump
> [SEVERS] into a /ServerRoot/Important.log
> the player stuff into /ServerRoot/Users.log
> the users stuff would have to probably detect a username and add it to
> a filter dynamically to automatically filter usernames and dump
> anything containing that kind of string into Users.log
> and finally possibly.
> /ServerRoot/plugins/PluginFolder/Plugin.log
>
>
> This would be quite an elegant solution if at all possible but I will
> look into seeing how much of this i can do myself once i get home. But
> i like good logs and i cannot lie. :D
>
>
>
> On Tue, Jan 10, 2012 at 9:15 AM, John Van Ostrand < john at netdirect.ca
> > wrote:
>
>
> Hi Colin,
>
> Although it matters how you want to split the minecraft log, perl was
> meant for this very purpose. You might also look at logrotate if you
> want to split on date/size. It does it automatically, just create a
> file in /etc/logrotate.d.
>
> If you want to split lines you may be able to do so depending on how
> minecraft writes logs.
>
> It may be using syslog functionality so modifying /etc/rsyslog.conf
> may reduce the amount of stuff in the file or be able to split it
> based on log level (info, debug, crit, emerg.)
>
> If it is using syslog and because of how it's using syslog I can't
> split it the way I want I write a filter and configure syslog to pipe
> to it through a named pipe.
>
> If you can't control how it writes logs then you'll have to manually
> process the file on regular intervals. Logrotate does this on size or
> date easily.
>
> A perl script like this is a start
>
> #!/usr/bin/perl
> #
> # Filter log file
> #
> # usage: myscript logfile
>
> while (<>) {
> # match text in lines
> if (/failed login/) {
> # Do something with it
> elsif (/new user/) {
> # do something else
> }
> }
>
> Awk is also built with this type of functionality, but perl is as well
> suited and has much more flexibility.
>
> To learn perl try http://learn.perl.org/ . But you may want to skip
> the "installing" section since perl is installed on almost everything.
>
>
>
>
>
> ----- Original Message -----
> > Well since i started this thread i didnt really expect it to take on
> > a
> > life of its own so i almost feel like this is off topic at this
> > point.
> > (dont poke the devs' :P) Anyways bash script looks like it would be
> > lovely a current application that im looking to try and write is
> > something to split a monolithic .log file for my minecraft server im
> > guessing it would require grep and some other such stuff but all of
> > this massive central log is just ugly and painful to sort through.
> > Server isnt even hugely popular yet and its already an 11mb log
> > file.
> > Its fairly important that I can search through it for keywords and
> > retain the logs for some time to help sort out disputes and track
> > staff abuse.
> >
> >
> > This isnt the only application that i've needed through my time on
> > computers but this is the one i need right now.
> >
> >
> > On Mon, Jan 9, 2012 at 11:56 AM, Paul Nijjar < paul_nijjar at yahoo.ca
> > >
> > wrote:
> >
> >
> >
> > On Sun, Jan 08, 2012 at 12:19:25PM -0500, Darcy Casselman wrote:
> > > A gentleman named Albert O'Connor is interested in starting up
> > > WatPy,
> > > a local Python user group. He's tentatively scheduled Wednesday,
> > > February 8 as the date of an initial organizational meeting. I
> > > don't
> > > have any other details, but I'll let him know there's interest
> > > here.
> >
> > It may be helpful to distinguish "User Group" from "Beginner
> > tutorial/course". That has been my experience with KWLUG, anyways.
> >
> > As some of you know I used the KW Freeskool infrastructure for the
> > Linux Literacy course. That may not reach the demographics you want
> > (and you may not want to be associated with their political
> > affiliation) but they can offer resources if you are interested.
> > http://kwfreeskool.wordpress.com
> >
> > I think both Dive into Python and Learn Python the Hard Way lack
> > sufficient exercises to be really good training materials, so you
> > will
> > have to come up with your own. An interesting mix might be to use
> > one
> > of these books in conjunction with the Project Euler
> > ( http://projecteuler.net ) series of problems.
>
>
> >
> > Another book that looks promising is "Building Skills in Python":
> > http://www.itmaybeahack.com/homepage/books/python.html
> >
> > You could also turn to open courseware. Here is a StackExchange
> > post with some good resources:
> >
> > http://stackoverflow.com/questions/4463098/where-can-i-find-a-good-online-python-course
> >
> > All of this depends on your audience. It makes a big difference
> > whether you are trying to get experienced programmers up to speed in
> > a
> > language, or whether your are trying to get people to learn
> > programming with Python as the vehicle.
> >
> > Sorry for the wall of text.
> >
> > - Paul
> >
> >
> >
> >
> > >
> > > Darcy.
> > >
> > > On Sun, Jan 8, 2012 at 12:09 PM, Michael Savage <
> > > msavage at golden.net
> > > > wrote:
> > > > I am interested.
> > > > Mike
> > > >
> > > >
> > > > On 08/01/2012 3:33 AM, Bob Jonkman wrote:
> > > >>
> > > >> Charles McColm and I were recently bitten by the
> > > >> "We-want-to-be-Python-programmers" bug. On Friday at Ubuntu
> > > >> Hour
> > > >> we met a
> > > >> fellow from London (ON). Sadly, my memory has lost its parity
> > > >> bits and I
> > > >> don't recall his name. Maybe "Mars" on the Ubuntu IRC channel?
> > > >> Anyway, he's
> > > >> a professional Python programmer, and he said he might be
> > > >> willing
> > > >> to come
> > > >> back to guide some of us newbies towards a life of Pythony
> > > >> goodness. He
> > > >> suggested we start with Mark Pilgrim's "Dive Into Python" or
> > > >> Zed
> > > >> A. Shaw's
> > > >> "Learn Python The Hard Way".
> > > >>
> > > >> Is there interest in learning Python within the ranks of KWLUG?
> > > >> All we
> > > >> need is a project to work on... The KWPUG group seems to be
> > > >> defunct, but we
> > > >> can strike out on our own.
> > > >>
> > > >> --Bob.
> > > >>
> > > >> Dive Into Python http://www.diveintopython.net/
> > > >> Learn Python The Hard Way http://learnpythonthehardway.org/
> > > >> KWPUG http://kwpug.org
> > > >> Ubuntu Canada IRC channel irc:// irc.freenode.net/#ubuntu-ca or
>
>
> > > >> http://webchat.freenode.net/?channels=ubuntu-ca
> > > >>
> > > >>
> > > >>
> > > >> John Van Ostrand < john at netdirect.ca > wrote:
> > > >>
> > > >> Hi Colon,
> > > >>
> > > >> Scripting in Linux relies heavily on the vast array of programs
> > > >> either directly or indirectly.
> > > >>
> > > >> Bash is the command shell and is commonly used for scripting,
> > > >> especially in system start up. It uses external programs
> > > >> heavily.
> > > >> I
> > > >> usually start with bash and if it proves poor for the task I
> > > >> move
> > > >> to
> > > >> perl by massaging the script.
> > > >>
> > > >> Perl is a very flexible and powerful scripting language and it
> > > >> borrows a lot of syntax from bash, awk and grep. So learning
> > > >> bash
> > > >> helps with perl. It works well for parsing text files.
> > > >>
> > > >> Python seems to be one of the more recently popular scripting.
> > > >> With
> > > >> add-ons it's used for Fedora admin Gui programs. It also
> > > >> handles
> > > >> threading well and has lots of other add-on modules.
> > > >>
> > > >> There are lots of other options but these are the common
> > > >> workhorse
> > > >> apps for system-level scripting.
> > > >> Bash and the common linux commands are worth getting familiar
> > > >> with.
> > > >> Regular expressions are also commonly handy. These are the
> > > >> basics
> > > >> that I findf handy all the time.
> > > >>
> > > >> What are you trying to do?
> > > >>
> > > >> ------------------------------------------------------------------------
> > > >> *From*: kwlug-disc-bounces at kwlug.org
> > > >> *To*: KWLug Discussion
> > > >> *Sent*: Sat Jan 07 18:48:02 2012
> > > >> *Subject*: [kwlug-disc] Scripting in Linux and unit.d
> > > >>
> > > >> I'm not a programmer but I do understand some fundamentals and
> > > >> looking to learn some scripting so that I can automate some
> > > >> tasks.
> > > >> What would be a good way to learn this stuff.
> > > >>
> > > >>
> > > >> _______________________________________________
> > > >> kwlug-disc mailing list
> > > >> kwlug-disc at kwlug.org
> > > >> http://kwlug.org/mailman/listinfo/kwlug-disc_kwlug.org
> > > >>
> > > >
> > > >
> > > > _______________________________________________
> > > > kwlug-disc mailing list
> > > > kwlug-disc at kwlug.org
> > > > http://kwlug.org/mailman/listinfo/kwlug-disc_kwlug.org
> > >
> > > _______________________________________________
> > > kwlug-disc mailing list
> > > kwlug-disc at kwlug.org
> > > http://kwlug.org/mailman/listinfo/kwlug-disc_kwlug.org
> >
> > --
> > http://pnijjar.freeshell.org
> >
> >
> > - Paul
> >
> >
> >
> > _______________________________________________
> > kwlug-disc mailing list
> > kwlug-disc at kwlug.org
> > http://kwlug.org/mailman/listinfo/kwlug-disc_kwlug.org
> >
> >
> > _______________________________________________
> > kwlug-disc mailing list
> > kwlug-disc at kwlug.org
> > http://kwlug.org/mailman/listinfo/kwlug-disc_kwlug.org
>
> --
> John Van Ostrand
> CTO, co-CEO
> Net Direct Inc.
> 564 Weber St. N. Unit 12, Waterloo, ON N2L 5C6
> Ph: 866-883-1172 x5102
> Fx: 519-883-8533
>
> Linux Solutions / IBM Hardware
>
>
>
>
> _______________________________________________
> kwlug-disc mailing list
> kwlug-disc at kwlug.org
> http://kwlug.org/mailman/listinfo/kwlug-disc_kwlug.org
>
>
> _______________________________________________
> kwlug-disc mailing list
> kwlug-disc at kwlug.org
> http://kwlug.org/mailman/listinfo/kwlug-disc_kwlug.org
--
John Van Ostrand
CTO, co-CEO
Net Direct Inc.
564 Weber St. N. Unit 12, Waterloo, ON N2L 5C6
Ph: 866-883-1172 x5102
Fx: 519-883-8533
Linux Solutions / IBM Hardware
More information about the kwlug-disc
mailing list