[vz-dev] Middleware Absturz

[Tom Weber]

Hei,

Thorben, du hattest mit deiner Vermutung Recht, unten ein Auszug aus dmesg
mit zig Kills.

Frage wäre: Was kann ich tun? Aufrüsten von Sheeva auf einen PC möchte ich
nicht so gerne, ich könnte es mit meinem NAS probieren (hat aber weniger
RAM)

Ich versuche jetzt erst mal, die my.cnf anzupassen...CHECK TABLE lief
übrigens brav durch.

Grüße,
Tom

[80949.330931] Out of memory: kill process 741 (mysqld_safe) score 49519 or
a child
[80949.352194] postgres invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[80949.363792] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[80949.373066] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[80949.647436] Out of memory: kill process 13105 (apache2) score 26792 or a
child
[82834.410512] mysqld invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[82834.421694] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[82834.430898] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[82834.705770] Out of memory: kill process 741 (mysqld_safe) score 48065 or
a child
[82835.405992] postgres invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[82835.417526] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[82835.426733] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[82835.701761] Out of memory: kill process 13563 (apache2) score 18661 or a
child
[94217.345931] postgres invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[94217.357471] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[94217.366676] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[94217.641382] Out of memory: kill process 15366 (apache2) score 31078 or a
child
[98309.566364] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[98309.577718] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[98309.586925] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[98309.861631] Out of memory: kill process 15880 (apache2) score 16692 or a
child
[98713.113644] apache2 invoked oom-killer: gfp_mask=0x200da, order=0,
oom_adj=0
[98713.125025] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[98713.134233] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[98713.370815] Out of memory: kill process 16107 (apache2) score 34086 or a
child
[99119.875929] startpar invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[99119.887556] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[99119.896758] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[99120.170186] Out of memory: kill process 15947 (apache2) score 17460 or a
child
[103476.365668] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[103476.377185] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[103476.386471] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[103476.663429] Out of memory: kill process 16781 (apache2) score 25603 or a
child
[103662.236340] postgres invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[103662.248039] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[103662.257331] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[103662.534118] Out of memory: kill process 16688 (apache2) score 16340 or a
child
[104543.725723] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[104543.737249] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[104543.746546] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[104544.023195] Out of memory: kill process 16869 (apache2) score 35752 or a
child
[104950.191145] apache2 invoked oom-killer: gfp_mask=0x200da, order=0,
oom_adj=0
[104950.202691] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[104950.211984] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[104950.452018] Out of memory: kill process 16956 (apache2) score 35431 or a
child
[105146.656268] startpar invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[105146.667967] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[105146.677265] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[105146.955291] Out of memory: kill process 741 (mysqld_safe) score 15195 or
a child
[105147.066661] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[105147.078199] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[105147.087488] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[105147.365192] Out of memory: kill process 16767 (apache2) score 14324 or a
child
[105767.338436] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[105767.349975] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[105767.359268] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[105767.635978] Out of memory: kill process 17063 (mysqld) score 75891 or a
child
[105767.658794] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[105767.670409] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[105767.679722] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[105767.956167] Out of memory: kill process 17138 (apache2) score 33387 or a
child
[105954.190698] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[105954.202335] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[105954.211628] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[105954.488162] Out of memory: kill process 17202 (mysqld) score 75837 or a
child
[110520.761220] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[110520.772753] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[110520.782092] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[110521.059211] Out of memory: kill process 17282 (mysqld) score 35407 or a
child
[110522.893832] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[110522.905447] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[110522.914748] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[110523.192367] Out of memory: kill process 17862 (apache2) score 17461 or a
child
[110703.251869] vzlogger invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[110703.263587] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[110703.272885] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[110703.550420] Out of memory: kill process 17977 (mysqld) score 54204 or a
child
[110703.569239] vzlogger invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[110703.580969] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[110703.590261] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[110703.867763] Out of memory: kill process 17885 (apache2) score 36907 or a
child
[112208.436100] vzlogger invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[112208.447801] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[112208.457195] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[112208.734018] Out of memory: kill process 18069 (mysqld) score 63398 or a
child
[112225.022844] mysqld invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[112225.034213] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[112225.043505] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[112225.331152] Out of memory: kill process 18335 (mysqld) score 34923 or a
child
[112225.685964] init invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[112225.696966] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[112225.706264] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[112225.982693] Out of memory: kill process 18290 (apache2) score 34921 or a
child
[112406.933561] vzlogger invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[112406.945280] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[112406.954691] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[112407.231207] Out of memory: kill process 18378 (mysqld) score 75848 or a
child
[112407.386128] init invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[112407.397125] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[112407.406415] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[112407.683077] Out of memory: kill process 18299 (apache2) score 32486 or a
child
[112597.466114] vzlogger invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[112597.477827] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[112597.487222] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[112597.763538] Out of memory: kill process 18443 (mysqld) score 63416 or a
child
[112597.779160] vzlogger invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[112597.790878] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[112597.800270] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[112598.076824] Out of memory: kill process 18431 (apache2) score 32937 or a
child
[112799.955749] apache2 invoked oom-killer: gfp_mask=0x200da, order=0,
oom_adj=0
[112799.967274] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[112799.976564] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[112800.216089] Out of memory: kill process 18545 (mysqld) score 54257 or a
child
[114239.366055] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[114239.377591] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[114239.386887] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[114239.663753] Out of memory: kill process 18720 (apache2) score 36134 or a
child
[114440.696547] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[114440.708080] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[114440.717377] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[114440.993876] Out of memory: kill process 18610 (mysqld) score 33156 or a
child
[114441.008136] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[114441.019682] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[114441.028974] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[114441.305612] Out of memory: kill process 18737 (apache2) score 11738 or a
child
[114624.187178] apache2 invoked oom-killer: gfp_mask=0x200da, order=0,
oom_adj=0
[114624.198707] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[114624.207998] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[114624.447369] Out of memory: kill process 18751 (mysqld) score 54256 or a
child
[119867.567690] apache2 invoked oom-killer: gfp_mask=0x200da, order=0,
oom_adj=0
[119867.579226] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[119867.588520] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[119867.827922] Out of memory: kill process 18817 (mysqld) score 35330 or a
child
[119876.460350] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[119876.471880] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[119876.481171] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[119876.758751] Out of memory: kill process 19018 (apache2) score 35307 or a
child
[120064.736111] postgres invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[120064.747817] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[120064.757106] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[120065.034629] Out of memory: kill process 19051 (mysqld) score 31503 or a
child
[120074.221272] apache2 invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[120074.233204] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[120074.242497] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[120074.519017] Out of memory: kill process 19036 (apache2) score 31145 or a
child
[120248.997055] vzlogger invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[120249.008756] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[120249.018049] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[120249.295585] Out of memory: kill process 19118 (mysqld) score 31516 or a
child
[120255.667962] postgres invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0
[120255.679668] [<c002df78>] (unwind_backtrace+0x0/0xdc) from [<c00a25fc>]
(oom_kill_process+0x5c/0x1d8)
[120255.688972] [<c00a25fc>] (oom_kill_process+0x5c/0x1d8) from [<c00a2bac>]
(__out_of_memory+0x148/0x16c)
[120255.966602] Out of memory: kill process 19002 (apache2) score 17109 or a
child

-----Ursprüngliche Nachricht-----
Von: volkszaehler-dev-bounces at lists.volkszaehler.org
[mailto:volkszaehler-dev-bounces at lists.volkszaehler.org] Im Auftrag von
Thorben Thuermer
Gesendet: Dienstag, 10. April 2012 13:59
An: volkszaehler.org
Betreff: Re: [vz-dev] Middleware Absturz

On Tue, 10 Apr 2012 09:08:24 +0200 "Tom Weber" <tom.weber at gmx.de> wrote:
> Hi,
> 
> ja, in den access.log sehe ich alle POSTS und auch die GETS - auch die 
> der Zeiten, wo die DB nicht mehr gefüllt wurde. Also scheinen die 
> requests durchgereicht zu werden.

kannst du die middleware-antworten denn sehen (manueller aufruf, vzlogger
mit hohem debug-level, notfalls traffic-sniffer) und sind die ok?

> Ein OPTIMIZE geht auf allen Tabellen bis auf DATA. Da kommt es nach 
> einigen Minuten zu:
> ERROR 2013 (HY000): Lost connection to MySQL server during query

schaue mal in dmesg, vermutlich wurde mysql aufgrund von speichermangel
gekillt...
("kernel: [...] Out of memory: Kill process ... (mysqld)") das koennte auch
deine anderen probleme erklaeren.

und: optimize ist NICHT noetig/sinnvoll...
das entfernt nach loeschoperationen freigewordene ungenutzte bereiche aus
der tabelle - aber volkszaehler loescht eigentlich eh nie was.
wenn check table sagt die tabelle ist ok, reicht das eigentlich, vielleicht
noch ein repair hinterher...

> Leider auch schon ca. 2 Mill Datensätze ....

kann gut sein, dass sich da irgendwann die grenzen einer so kleinen hardware
zeigen...
(also, fuer den normalen betrieb reicht es, aber nicht fuer operationen  auf
so grossen tabellen als ganzes.) notfalls kann man die tabelle vermutlich
noch auf eine groesseres system (gleicher endianess?) kopieren und dort
reparieren. (auch wenn mysql nicht garantiert das das funktioniert.)

> Tom

- Thorben

> -----Ursprüngliche Nachricht-----
> Von: volkszaehler-dev-bounces at lists.volkszaehler.org
> [mailto:volkszaehler-dev-bounces at lists.volkszaehler.org] Im Auftrag 
> von Thorben Thuermer
> Gesendet: Montag, 9. April 2012 22:02
> An: volkszaehler.org
> Betreff: Re: [vz-dev] Middleware Absturz
> 
> On Mon, 9 Apr 2012 21:41:44 +0200 "Tom Weber" <tom.weber at gmx.de> wrote:
> > ich habe bei meiner Sheeva Middleware immer wieder das Problem, dass 
> > sie sich nach einigen Tagen oder Stunden verabschiedet.
> > Es findet dann keinerlei Aufzeichnung mehr statt, weder s0 noch
vzlogger.
> > Bizarrerweise kann ich aber die middleware.php aufrufen, apache 
> > scheint also zu laufen. Auch die error.log zeigt in meinen Augen 
> > keine Auffälligkeiten....
> > 
> > Wie kann ich dem Fehler auf die Schliche kommen?
> 
> also, die middleware als solche kann ja nicht "abstuerzen", zumal es 
> sich um ein php-script handelt, das vom webserver fuer fuer jeden 
> eingehenden request neu ausgefuehrt wird.
> 
> die fragen die zu klaeren sind sind im prinzip:
> * erzeugen der/die client/s noch requests
> * erreichen diese requests den webserver
> * verarbeitet der webserver diese
> * treten bei der verarbeitung probleme auf
> 
> du sagst, manuell abgesetzte requests funktionieren noch?
> betrifft das nur das abrufen von daten, oder auch das eintragen (zB 
> auf einem dafuer erzeugten test-kanal)?
> 
> du sagst im error-log steht nichts...
> wie sieht es dann mit access.log aus? da sollten die requests in jedem 
> fall auftauchen, wenn sie gesendet werden und den server erreichen.
> 
> im zweifelsfall mit wireshark etc. pruefen, ob noch requests ueber's 
> netz gehen.
> 
> was ich auch lohnt zu pruefen ist, ob die dahinterliegende 
> mysql-datenbank ggfs. probleme hat - das ist auch ein haeufiges problem.
> (show processlist - haengen ggfs mysql-anfragen (platte voll?),  oder 
> mal ein check/repair table.)
> 
> > Hilfe wäre Klasse,
> > Tom
> 
> - Thorben
>