vegdZdZddlZddlZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZdd lmZmZdd lmZdd lmZdd lmZd dlmZd dlmZddlmZddlm Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(GddeZ)e*dkrce&ej+dej, e)-dS#e $rZ.e/e.e0d dZ.[.wwxYwdS)zd This module contains SSA classes for sending e-mails with report and sending reports to ClickHouse z ssa.modulesN) EmailMessage)partial)OptionalAny) gethostname)get_admin_email)SessionResponse) HTTPAdapter)RequestException)Retry)Common) DecisionMaker) stat_server)SSAError)Mailerrender_report_table) read_sys_id sentry_init duration_cast format_dateceZdZdZdedfdeffd Zddeddfd Zd ede e fd Z de efd Z de efd Z d edefdZededefdZdedefdZdedefdZxZS)StatisticsSenderz4 Send report to ClickHouse and over e-mails https://z /api/clos-ssa ch_endpointc:ttjd|_|jdt t|_||_ t|_ tdtdgtgdd}t|}t|_|jd|t%|jjd |j_dS) N stat_senderzStatisticsSender enabled: %sPOST)iii)totalallowed_methodsstatus_forcelistbackoff_factor) max_retriesr )timeout)super__init__logging getLoggerloggerinfo __package__rsys_idrr mail_senderr frozensetr r sessionmountrrequest)selfr retry_confadapter __class__s N/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/modules/stat_sender.pyr*zStatisticsSender.__init__*s ' 66  7EEE!mm &!88+4fX+>+>,5ooo,F,F*+--- *555yy  :w///&t|';RHHH Nreportreturnc| t}|jr||||dS)zi Send given report over e-mail If no report given, get it from DecisionMaker API N)rget_json_reportsummary_notification_enabled email_reportclickhouse_report)r6r<s r:sendzStatisticsSender.send;sW >"__4466F  , &   f % % % v&&&&&r; report_viewc|}|r|drt|\}}|j|dt |dt |}|tj |dd|dd||d d|dd |j ||Sd Sd S) z4 Create and send e-mail with report domains ssa_reportdate) recipienttemplaterHhostnamehtmljsonreport_z.json)subtypefilenamerLz.htmlN) get_mail_recipientgetrr1_messagerradd_attachmentrMdumps_send)r6rDmail_to report_tablemailmsgs r:rAzStatisticsSender.email_reportGs/))++  {y11 !4[!A!A L$"++!% V!455$ ,C   tz+66'-(L+f2E(L(L(L  N N N   |'-(L+f2E(L(L(L  N N N   " "3 ' ' 'J%    r;c |p tS#t$r3}|jdt |Yd}~dSd}~wwxYw)z Retrieve a recipient's e-mail: 1. get address from a wmt-api utility 2. if command failed or address is empty, get address of server admin zget_admin_email failed with: %sN)wmt_api_report_emailr Exceptionr-errorstr)r6es r:rQz#StatisticsSender.get_mail_recipientasy  I,,..C/2C2C C I I I K  ?Q H H H H H H H H H Is!$ A!(AA!cpd} tjd|dgdddj}nX#tjt t tf$r3}|j dt|Yd}~dSd}~wwxYw tj | d d S#tj$r3}|j d t|Yd}~dSd}~wt $r|j d |YdSwxYw) zD Retrieve a recipient's e-mail address from WMT API zwmt-apiz&/usr/share/web-monitoring-tool/wmtbin/z --config-getT)checktextcapture_outputzwmt-api utility failed: %sNconfig report_emailz!wmt-api returned invalid json: %sz(wmt-api returned unexpected response: %s) subprocessrunstdoutstripCalledProcessErrorAttributeErrorOSError ValueErrorr-r^r_rMloadsrRJSONDecodeError)r6_util api_responser`s r:r\z%StatisticsSender.wmt_api_report_emailms %>A%AA!# %%%&,EEGG L -~w    K  :CFF C C C FFFFF   ,:l++//99==nMM M# & & & K  A!!ff & & & & & & & & & , , , K  H* , , , , , , ,s349"B(B  B9C D5(D  (D54D5c |dr|jd|j |j|j|||}nV#t$rI}|j d|d|jitdt||d}~wwxYw| |S|jd d S) z+ Send report to ClickHouse rFzSending POST request to %s)rMzPOST failed with %sendpointextraz'Failed to POST data to SSA API server: Nz*Report is empty, not sending to ClickHouseF) rRr-r.rr3post_ch_packclickhouse_formatr r^rr__process_response)r6rDrespr`s r:rBz"StatisticsSender.clickhouse_reports5 ??9 % %  K  9!- / / / O|(()9 **;77A9A9(::# O O O !!"7)3T5E(F"HHHFc!ffFFHHMNO O ))$// / K  I J J J5sAA?? C AC  Cvaluec"t|S)z< Pack given value into data field of a dict )data)dict)r|s r:rxzStatisticsSender._ch_packs r;original_reportc ft}|dD]}||j|d|d|d|dd|dDd|S) zd Format local report for sending to ClickHouse (required structures differ) rFname slow_urls slow_reqs total_reqsc g|]p}|d|dt|dt|dddqS)rreqs_numaverage_duration correlationr)urlcount_requests avg_durationr)rRrfloat).0us r: z6StatisticsSender.clickhouse_format..s !uuV}}*+%% *;*;(5EE"455)7)7',QUU=!-D-D'E'E r;urls) system_iddomaincount_slow_urlscount_slow_requeststotal_requestsdetails)listrRappendr0)r6r ch_reportrs r:ryz"StatisticsSender.clickhouse_formats FF %)))44  F   ![ **V,,#)::k#:#:'-zz+'>'>"(**\":": &zz&11      r;responsec|js1|jd|j|jd|jidS|jd|j|j|j|}|ddkr'|jd|dd |idS|jd d S) z Check received response :param response: a requests.Response object :return: True in case of success, False otherwise z&Unable to connect to server with %s:%s resp_textruFz[%s:%s] Response received %sstatusokz Received response with status %srzSent to ClickHouse successfullyT) rr-r^ status_codereasonrcr.rrM)r6rresults r:rzz"StatisticsSender._process_responses { + K  F&2HO%0(-$@  B B B5 K  ;%18?%\ + + + ( t # # K  @$X.z66J  L L L5 :;;;tr;)N)__name__ __module__ __qualname____doc__rr_r*rrCrrrArQr\boolrB staticmethodrrxrryr rz __classcell__)r9s@r:rr%s+P[*O*O*OIICIIIIII" ' '4 '4 ' ' ' ',1G4 IHSM I I I I,hsm,,,,2Td(     \ $2(tr;r__main__zstat_sender_standalone.log)rPlevel)1rr/rMr+rg email.messager functoolsrtypingrrsocketrclcommon.cpapirrequestsr r requests.adaptersr requests.exceptionsr $requests.packages.urllib3.util.retryr commonrdecision_makerrinternal.constantsrinternal.exceptionsrinternal.mailerrrinternal.utilsrrrrrr basicConfigINFOrCexcprint SystemExitr;r:rsU &&&&&& ******&&&&&&&&))))))000000666666)))))),,,,,,******99999999lllllvlll^ zKMMMG!=%l,,,,!!!!!  c jmm s-C C+C&&C+