<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>开发工具 on heyaohua's Blog</title><link>https://blog.heyaohua.com/categories/%E5%BC%80%E5%8F%91%E5%B7%A5%E5%85%B7/</link><description>Recent content in 开发工具 on heyaohua's Blog</description><image><title>heyaohua's Blog</title><url>https://blog.heyaohua.com/og-image.png</url><link>https://blog.heyaohua.com/og-image.png</link></image><generator>Hugo</generator><language>zh-cn</language><lastBuildDate>Fri, 26 Sep 2025 14:00:00 +0800</lastBuildDate><atom:link href="https://blog.heyaohua.com/categories/%E5%BC%80%E5%8F%91%E5%B7%A5%E5%85%B7/index.xml" rel="self" type="application/rss+xml"/><item><title>淘宝自动化框架选择方案</title><link>https://blog.heyaohua.com/posts/2025/09/taobao-automation-framework/</link><pubDate>Fri, 26 Sep 2025 14:00:00 +0800</pubDate><guid>https://blog.heyaohua.com/posts/2025/09/taobao-automation-framework/</guid><description>国产框架，中文文档完善</description><content:encoded><![CDATA[<h1 id="淘宝自动化框架选择方案">淘宝自动化框架选择方案</h1>
<h2 id="-推荐方案drissionpage--现有架构">🎯 推荐方案：DrissionPage + 现有架构</h2>
<h3 id="为什么选择-drissionpage">为什么选择 DrissionPage？</h3>
<ol>
<li><strong>专为中国网站设计</strong></li>
<li>针对淘宝、京东等电商网站优化</li>
<li>内置常见反爬虫机制绕过</li>
<li></li>
</ol>
<p>国产框架，中文文档完善</p>
<ol start="5">
<li></li>
</ol>
<p><strong>与现有架构完美融合</strong></p>
<ol start="6">
<li>可以直接使用现有的 requests session</li>
<li>支持与 mitmproxy 代理集成</li>
<li></li>
</ol>
<p>兼容现有的数据处理管道</p>
<ol start="9">
<li></li>
</ol>
<p><strong>性能与易用性并存</strong></p>
<ol start="10">
<li>基于 Chromium 内核，性能优秀</li>
<li>API 设计简洁直观</li>
<li>支持页面模式和 requests 模式切换</li>
</ol>
<h2 id="-框架对比分析">📊 框架对比分析</h2>
<table>
  <thead>
      <tr>
          <th>特性</th>
          <th>DrissionPage</th>
          <th>Playwright</th>
          <th>Selenium</th>
          <th>Requests-HTML</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>性能</strong></td>
          <td>很快</td>
          <td>最快</td>
          <td>中等</td>
          <td>快</td>
      </tr>
      <tr>
          <td><strong>反爬虫能力</strong></td>
          <td>优秀</td>
          <td>优秀</td>
          <td>一般</td>
          <td>较弱</td>
      </tr>
      <tr>
          <td><strong>淘宝适配</strong></td>
          <td>优秀</td>
          <td>好</td>
          <td>一般</td>
          <td>较弱</td>
      </tr>
      <tr>
          <td><strong>学习成本</strong></td>
          <td>低</td>
          <td>中</td>
          <td>中</td>
          <td>低</td>
      </tr>
      <tr>
          <td><strong>中文文档</strong></td>
          <td>优秀</td>
          <td>一般</td>
          <td>好</td>
          <td>一般</td>
      </tr>
      <tr>
          <td><strong>社区支持</strong></td>
          <td>活跃</td>
          <td>活跃</td>
          <td>最大</td>
          <td>较小</td>
      </tr>
  </tbody>
</table>
<h2 id="-技术实施路线">🛠️ 技术实施路线</h2>
<h3 id="阶段一环境准备">阶段一：环境准备</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#6272a4"># 安装 DrissionPage</span>
</span></span><span style="display:flex;"><span>pip install DrissionPage
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#6272a4"># 安装备选方案（可选）</span>
</span></span><span style="display:flex;"><span>pip install playwright
</span></span><span style="display:flex;"><span>pip install selenium
</span></span></code></pre></div><h3 id="阶段二基础集成">阶段二：基础集成</h3>
<ol>
<li>创建 <code>TaobaoAutomator</code> 类</li>
<li>集成现有的代理服务器</li>
<li>实现基础的搜索和数据提取功能</li>
</ol>
<h3 id="阶段三高级功能">阶段三：高级功能</h3>
<ol>
<li>反爬虫策略优化</li>
<li>数据清洗和存储</li>
<li>错误处理和重试机制</li>
</ol>
<h3 id="阶段四性能优化">阶段四：性能优化</h3>
<ol>
<li>并发处理</li>
<li>资源管理</li>
<li>监控和日志</li>
</ol>
<h2 id="-备选方案">💡 备选方案</h2>
<h3 id="方案-a纯-playwright如果团队技术能力强">方案 A：纯 Playwright（如果团队技术能力强）</h3>
<ul>
<li>性能最佳</li>
<li>功能最全面</li>
<li>需要较多学习时间</li>
</ul>
<h3 id="方案-bselenium如果需要最大兼容性">方案 B：Selenium（如果需要最大兼容性）</h3>
<ul>
<li>社区资源最丰富</li>
<li>兼容性最好</li>
<li>性能相对较慢</li>
</ul>
<h3 id="方案-c混合方案">方案 C：混合方案</h3>
<ul>
<li>DrissionPage 处理复杂交互</li>
<li>requests 处理简单API调用</li>
<li>mitmproxy 处理数据截取</li>
</ul>
<h2 id="-具体实现示例">🎪 具体实现示例</h2>
<h3 id="drissionpage-基础用法">DrissionPage 基础用法</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#ff79c6">from</span> DrissionPage <span style="color:#ff79c6">import</span> ChromiumPage
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#6272a4"># 创建页面对象</span>
</span></span><span style="display:flex;"><span>page <span style="color:#ff79c6">=</span> ChromiumPage()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#6272a4"># 访问淘宝</span>
</span></span><span style="display:flex;"><span>page<span style="color:#ff79c6">.</span>get(<span style="color:#f1fa8c">&#39;https://www.taobao.com&#39;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#6272a4"># 搜索商品</span>
</span></span><span style="display:flex;"><span>search_box <span style="color:#ff79c6">=</span> page<span style="color:#ff79c6">.</span>ele(<span style="color:#f1fa8c">&#39;#q&#39;</span>)
</span></span><span style="display:flex;"><span>search_box<span style="color:#ff79c6">.</span>input(<span style="color:#f1fa8c">&#39;手机&#39;</span>)
</span></span><span style="display:flex;"><span>search_box<span style="color:#ff79c6">.</span>after()<span style="color:#ff79c6">.</span>click()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#6272a4"># 获取商品信息</span>
</span></span><span style="display:flex;"><span>products <span style="color:#ff79c6">=</span> page<span style="color:#ff79c6">.</span>eles(<span style="color:#f1fa8c">&#39;.item&#39;</span>)
</span></span><span style="display:flex;"><span><span style="color:#ff79c6">for</span> product <span style="color:#ff79c6">in</span> products:
</span></span><span style="display:flex;"><span>    title <span style="color:#ff79c6">=</span> product<span style="color:#ff79c6">.</span>ele(<span style="color:#f1fa8c">&#39;.title&#39;</span>)<span style="color:#ff79c6">.</span>text
</span></span><span style="display:flex;"><span>    price <span style="color:#ff79c6">=</span> product<span style="color:#ff79c6">.</span>ele(<span style="color:#f1fa8c">&#39;.price&#39;</span>)<span style="color:#ff79c6">.</span>text
</span></span><span style="display:flex;"><span>    <span style="color:#8be9fd;font-style:italic">print</span>(<span style="color:#f1fa8c">f</span><span style="color:#f1fa8c">&#34;</span><span style="color:#f1fa8c">{</span>title<span style="color:#f1fa8c">}</span><span style="color:#f1fa8c">: </span><span style="color:#f1fa8c">{</span>price<span style="color:#f1fa8c">}</span><span style="color:#f1fa8c">&#34;</span>)
</span></span></code></pre></div><h3 id="与现有架构集成">与现有架构集成</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#ff79c6">from</span> DrissionPage <span style="color:#ff79c6">import</span> ChromiumPage
</span></span><span style="display:flex;"><span><span style="color:#ff79c6">from</span> crawler.gateway.proxy_server <span style="color:#ff79c6">import</span> ProxyServer
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff79c6">class</span> <span style="color:#50fa7b">TaobaoAutomator</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">def</span> <span style="color:#50fa7b">__init__</span>(<span style="font-style:italic">self</span>):
</span></span><span style="display:flex;"><span>        <span style="color:#6272a4"># 启动代理服务器</span>
</span></span><span style="display:flex;"><span>        <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>proxy_server <span style="color:#ff79c6">=</span> ProxyServer()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#6272a4"># 配置 DrissionPage 使用代理</span>
</span></span><span style="display:flex;"><span>        <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>page <span style="color:#ff79c6">=</span> ChromiumPage()
</span></span><span style="display:flex;"><span>        <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>page<span style="color:#ff79c6">.</span>set<span style="color:#ff79c6">.</span>proxy(<span style="color:#f1fa8c">f</span><span style="color:#f1fa8c">&#39;127.0.0.1:</span><span style="color:#f1fa8c">{</span><span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>proxy_server<span style="color:#ff79c6">.</span>port<span style="color:#f1fa8c">}</span><span style="color:#f1fa8c">&#39;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">def</span> <span style="color:#50fa7b">search_products</span>(<span style="font-style:italic">self</span>, keyword):
</span></span><span style="display:flex;"><span>        <span style="color:#6272a4"># 实现搜索逻辑</span>
</span></span><span style="display:flex;"><span>        <span style="color:#ff79c6">pass</span>
</span></span></code></pre></div><h2 id="-技术要点">🔧 技术要点</h2>
<ol>
<li><strong>代理集成</strong>：确保自动化框架使用现有的代理服务器</li>
<li><strong>数据同步</strong>：截取的API数据与页面数据关联</li>
<li><strong>反爬虫</strong>：实现用户行为模拟和请求间隔控制</li>
<li><strong>错误处理</strong>：网络异常、页面变化等情况的处理</li>
</ol>
<h2 id="-预期效果">📈 预期效果</h2>
<ul>
<li><strong>开发效率提升 50%</strong>：相比从零开始</li>
<li><strong>数据质量提升</strong>：结合API和页面数据</li>
<li><strong>稳定性增强</strong>：多重反爬虫策略</li>
<li><strong>维护成本降低</strong>：统一的架构设计</li>
</ul>
]]></content:encoded></item><item><title>我用Python开发了一个淘宝图片搜索自动化系统</title><link>https://blog.heyaohua.com/posts/2025/05/taobao-image-search-automation/</link><pubDate>Mon, 26 May 2025 16:00:00 +0800</pubDate><guid>https://blog.heyaohua.com/posts/2025/05/taobao-image-search-automation/</guid><description>在电商时代，图片搜索已经成为用户发现商品的重要方式。作为开发者，我经常需要为客户批量搜索相似商品并生成报告。手动操作不仅效率低下，还容易出错。于是，我决定开发一个自动化系统来解决这个问题。</description><content:encoded><![CDATA[<p>在电商时代，图片搜索已经成为用户发现商品的重要方式。作为开发者，我经常需要为客户批量搜索相似商品并生成报告。手动操作不仅效率低下，还容易出错。于是，我决定开发一个自动化系统来解决这个问题。</p>
<h2 id="项目目标">项目目标</h2>
<ul>
<li>批量处理图片搜索</li>
<li>自动提取商品数据</li>
<li>生成包含图片的Excel报告</li>
<li>自动发送邮件通知</li>
<li>完整的错误处理和日志记录</li>
</ul>
<h2 id="技术选型">技术选型</h2>
<h3 id="自动化框架drissionpage">自动化框架：DrissionPage</h3>
<p>经过对比Selenium、Playwright等框架，我选择了DrissionPage：</p>
<ul>
<li>专为中国网站优化</li>
<li>反爬虫能力强</li>
<li>对淘宝等国内电商支持好</li>
</ul>
<h3 id="数据拦截mitmproxy">数据拦截：mitmproxy</h3>
<ul>
<li>能够拦截HTTPS流量</li>
<li>支持自定义插件</li>
<li>适合API数据提取</li>
</ul>
<h3 id="数据处理">数据处理</h3>
<ul>
<li>Pandas：数据处理</li>
<li>openpyxl：Excel操作</li>
<li>Pillow：图片处理</li>
</ul>
<h2 id="核心功能实现">核心功能实现</h2>
<h3 id="1-图片搜索自动化">1. 图片搜索自动化</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#ff79c6">def</span> <span style="color:#50fa7b">search_by_image</span>(<span style="font-style:italic">self</span>, image_path: <span style="color:#8be9fd;font-style:italic">str</span>):
</span></span><span style="display:flex;"><span>    <span style="color:#f1fa8c">&#34;&#34;&#34;图片搜索功能&#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#6272a4"># 1. 打开淘宝首页</span>
</span></span><span style="display:flex;"><span>    <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>browser<span style="color:#ff79c6">.</span>get(<span style="color:#f1fa8c">&#39;https://www.taobao.com&#39;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#6272a4"># 2. 点击搜同款按钮</span>
</span></span><span style="display:flex;"><span>    search_button <span style="color:#ff79c6">=</span> <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>browser<span style="color:#ff79c6">.</span>ele(<span style="color:#f1fa8c">&#39;css:.image-search-icon-wrapper&#39;</span>)
</span></span><span style="display:flex;"><span>    search_button<span style="color:#ff79c6">.</span>click()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#6272a4"># 3. 上传图片</span>
</span></span><span style="display:flex;"><span>    file_input <span style="color:#ff79c6">=</span> <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>browser<span style="color:#ff79c6">.</span>ele(<span style="color:#f1fa8c">&#39;css:#image-search-custom-file-input&#39;</span>)
</span></span><span style="display:flex;"><span>    file_input<span style="color:#ff79c6">.</span>input(image_path)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#6272a4"># 4. 等待上传完成并搜索</span>
</span></span><span style="display:flex;"><span>    <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>_wait_for_upload_complete()
</span></span><span style="display:flex;"><span>    search_btn <span style="color:#ff79c6">=</span> <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>browser<span style="color:#ff79c6">.</span>ele(<span style="color:#f1fa8c">&#39;css:#image-search-upload-button&#39;</span>)
</span></span><span style="display:flex;"><span>    search_btn<span style="color:#ff79c6">.</span>click()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#6272a4"># 5. 提取商品数据</span>
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">return</span> <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>_extract_products_from_page()
</span></span></code></pre></div><h3 id="2-数据拦截与提取">2. 数据拦截与提取</h3>
<p>通过mitmproxy拦截淘宝API响应，提取商品信息：</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#ff79c6">def</span> <span style="color:#50fa7b">response</span>(flow: http<span style="color:#ff79c6">.</span>HTTPFlow) <span style="color:#ff79c6">-&gt;</span> <span style="color:#ff79c6">None</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f1fa8c">&#34;&#34;&#34;拦截API响应&#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">if</span> <span style="color:#f1fa8c">&#39;h5api.m.taobao.com&#39;</span> <span style="color:#ff79c6">in</span> flow<span style="color:#ff79c6">.</span>request<span style="color:#ff79c6">.</span>pretty_url:
</span></span><span style="display:flex;"><span>        content <span style="color:#ff79c6">=</span> flow<span style="color:#ff79c6">.</span>response<span style="color:#ff79c6">.</span>text
</span></span><span style="display:flex;"><span>        <span style="color:#6272a4"># 解析JSONP响应，提取商品数据</span>
</span></span><span style="display:flex;"><span>        data <span style="color:#ff79c6">=</span> parse_jsonp_response(content)
</span></span><span style="display:flex;"><span>        save_to_file(data)
</span></span></code></pre></div><h3 id="3-excel报告生成">3. Excel报告生成</h3>
<p>生成多Sheet的Excel文件，包含压缩图片：</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#ff79c6">def</span> <span style="color:#50fa7b">generate_excel_report</span>(<span style="font-style:italic">self</span>, products_data):
</span></span><span style="display:flex;"><span>    <span style="color:#f1fa8c">&#34;&#34;&#34;生成Excel报告&#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    workbook <span style="color:#ff79c6">=</span> openpyxl<span style="color:#ff79c6">.</span>Workbook()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">for</span> sheet_data <span style="color:#ff79c6">in</span> products_data:
</span></span><span style="display:flex;"><span>        worksheet <span style="color:#ff79c6">=</span> workbook<span style="color:#ff79c6">.</span>create_sheet(sheet_data[<span style="color:#f1fa8c">&#39;name&#39;</span>])
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#6272a4"># 添加商品数据</span>
</span></span><span style="display:flex;"><span>        <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>add_product_data(worksheet, sheet_data[<span style="color:#f1fa8c">&#39;products&#39;</span>])
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#6272a4"># 下载并添加商品图片</span>
</span></span><span style="display:flex;"><span>        <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>add_product_images(worksheet, sheet_data[<span style="color:#f1fa8c">&#39;products&#39;</span>])
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    workbook<span style="color:#ff79c6">.</span>save(<span style="color:#f1fa8c">&#39;report.xlsx&#39;</span>)
</span></span></code></pre></div><h3 id="4-图片压缩优化">4. 图片压缩优化</h3>
<p>解决Excel文件过大的问题：</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#ff79c6">def</span> <span style="color:#50fa7b">compress_image</span>(<span style="font-style:italic">self</span>, image_path: <span style="color:#8be9fd;font-style:italic">str</span>) <span style="color:#ff79c6">-&gt;</span> <span style="color:#8be9fd;font-style:italic">str</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f1fa8c">&#34;&#34;&#34;智能图片压缩&#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">with</span> Image<span style="color:#ff79c6">.</span>open(image_path) <span style="color:#ff79c6">as</span> img:
</span></span><span style="display:flex;"><span>        <span style="color:#6272a4"># 调整尺寸到400x400</span>
</span></span><span style="display:flex;"><span>        <span style="color:#ff79c6">if</span> img<span style="color:#ff79c6">.</span>width <span style="color:#ff79c6">&gt;</span> <span style="color:#bd93f9">400</span> <span style="color:#ff79c6">or</span> img<span style="color:#ff79c6">.</span>height <span style="color:#ff79c6">&gt;</span> <span style="color:#bd93f9">400</span>:
</span></span><span style="display:flex;"><span>            img<span style="color:#ff79c6">.</span>thumbnail((<span style="color:#bd93f9">400</span>, <span style="color:#bd93f9">400</span>), Image<span style="color:#ff79c6">.</span>Resampling<span style="color:#ff79c6">.</span>LANCZOS)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#6272a4"># 压缩质量到40%</span>
</span></span><span style="display:flex;"><span>        img<span style="color:#ff79c6">.</span>save(compressed_path, <span style="color:#f1fa8c">&#34;JPEG&#34;</span>, quality<span style="color:#ff79c6">=</span><span style="color:#bd93f9">40</span>, optimize<span style="color:#ff79c6">=</span><span style="color:#ff79c6">True</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">return</span> compressed_path
</span></span></code></pre></div><h2 id="技术难点与解决方案">技术难点与解决方案</h2>
<h3 id="1-反爬虫对抗">1. 反爬虫对抗</h3>
<p><strong>问题</strong>：淘宝有完善的反爬虫机制</p>
<p><strong>解决方案</strong>：</p>
<ul>
<li>使用DrissionPage框架</li>
<li>设置随机延迟</li>
<li>模拟真实用户行为</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#6272a4"># 随机延迟模拟人类行为</span>
</span></span><span style="display:flex;"><span><span style="color:#ff79c6">import</span> random
</span></span><span style="display:flex;"><span>time<span style="color:#ff79c6">.</span>sleep(random<span style="color:#ff79c6">.</span>uniform(<span style="color:#bd93f9">1</span>, <span style="color:#bd93f9">3</span>))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#6272a4"># 滚动页面</span>
</span></span><span style="display:flex;"><span><span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>browser<span style="color:#ff79c6">.</span>scroll_to_bottom()
</span></span></code></pre></div><h3 id="2-图片上传处理">2. 图片上传处理</h3>
<p><strong>问题</strong>：淘宝使用隐藏的file input</p>
<p><strong>解决方案</strong>：</p>
<ul>
<li>使用JavaScript直接设置文件路径</li>
<li>监听上传进度事件</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span># 直接设置文件路径
</span></span><span style="display:flex;"><span>file_input = self.browser.ele(&#39;css:#image-search-custom-file-input&#39;)
</span></span><span style="display:flex;"><span>self.browser.run_js(f&#34;arguments[0].value = &#39;{image_path}&#39;&#34;, file_input)
</span></span></code></pre></div><h3 id="3-数据解析复杂性">3. 数据解析复杂性</h3>
<p><strong>问题</strong>：淘宝API返回JSONP格式，结构复杂</p>
<p><strong>解决方案</strong>：</p>
<ul>
<li>递归解析JSON结构</li>
<li>使用多种字段别名匹配</li>
<li>建立数据质量评分</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#ff79c6">def</span> <span style="color:#50fa7b">find_items_recursively</span>(<span style="font-style:italic">self</span>, obj):
</span></span><span style="display:flex;"><span>    <span style="color:#f1fa8c">&#34;&#34;&#34;递归查找商品数据&#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">if</span> <span style="color:#8be9fd;font-style:italic">isinstance</span>(obj, <span style="color:#8be9fd;font-style:italic">dict</span>) <span style="color:#ff79c6">and</span> <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>_is_product_item(obj):
</span></span><span style="display:flex;"><span>        <span style="color:#ff79c6">return</span> [<span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>_extract_product_info(obj)]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">if</span> <span style="color:#8be9fd;font-style:italic">isinstance</span>(obj, <span style="color:#8be9fd;font-style:italic">list</span>):
</span></span><span style="display:flex;"><span>        <span style="color:#ff79c6">for</span> item <span style="color:#ff79c6">in</span> obj:
</span></span><span style="display:flex;"><span>            result <span style="color:#ff79c6">=</span> <span style="font-style:italic">self</span><span style="color:#ff79c6">.</span>find_items_recursively(item)
</span></span><span style="display:flex;"><span>            <span style="color:#ff79c6">if</span> result:
</span></span><span style="display:flex;"><span>                <span style="color:#ff79c6">return</span> result
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">return</span> []
</span></span></code></pre></div><h2 id="项目成果">项目成果</h2>
<h3 id="功能实现">功能实现</h3>
<ul>
<li>✅ 批量图片搜索（15张图片/批次）</li>
<li>✅ 自动数据提取和解析</li>
<li>✅ 多Sheet Excel报告生成</li>
<li>✅ 邮件自动发送</li>
<li>✅ 数据自动清理</li>
</ul>
<h3 id="性能指标">性能指标</h3>
<ul>
<li><strong>处理速度</strong>：15张图片约3分钟</li>
<li><strong>文件大小</strong>：从326MB压缩到16MB</li>
<li><strong>成功率</strong>：95%以上</li>
<li><strong>稳定性</strong>：支持错误重试</li>
</ul>
<h3 id="用户体验">用户体验</h3>
<ul>
<li><strong>一键运行</strong>：<code>python run.py</code></li>
<li><strong>配置简单</strong>：只需配置邮件信息</li>
<li><strong>日志详细</strong>：完整的执行日志</li>
</ul>
<h2 id="项目结构">项目结构</h2>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>taobao-search/
</span></span><span style="display:flex;"><span>├── run.py                    # 主启动脚本
</span></span><span style="display:flex;"><span>├── src/                      # 源代码
</span></span><span style="display:flex;"><span>│   ├── automation/           # 自动化模块
</span></span><span style="display:flex;"><span>│   ├── email/               # 邮件服务
</span></span><span style="display:flex;"><span>│   ├── excel/               # Excel处理
</span></span><span style="display:flex;"><span>│   └── workflow/            # 工作流程
</span></span><span style="display:flex;"><span>├── config/                  # 配置文件
</span></span><span style="display:flex;"><span>├── IMG_LIST/                # 图片目录
</span></span><span style="display:flex;"><span>└── data/                    # 数据目录
</span></span></code></pre></div><h2 id="使用方法">使用方法</h2>
<ol>
<li><strong>准备图片</strong>：将图片放入<code>IMG_LIST</code>目录</li>
<li><strong>配置邮件</strong>：编辑<code>config/email_config.json</code></li>
<li><strong>一键运行</strong>：<code>python run.py</code></li>
<li><strong>查看结果</strong>：Excel文件在<code>data/exports/</code>目录</li>
</ol>
<h2 id="技术总结">技术总结</h2>
<h3 id="收获">收获</h3>
<ol>
<li><strong>自动化框架选择</strong>：DrissionPage在反爬虫方面表现优秀</li>
<li><strong>数据拦截技术</strong>：mitmproxy是API数据提取的有效方案</li>
<li><strong>图片处理优化</strong>：合理的压缩策略能显著减小文件大小</li>
<li><strong>工作流程设计</strong>：模块化设计便于维护和扩展</li>
</ol>
<h3 id="价值">价值</h3>
<ul>
<li><strong>效率提升</strong>：从手动操作到全自动化，效率提升10倍</li>
<li><strong>质量保证</strong>：自动化处理减少人为错误</li>
<li><strong>可扩展性</strong>：模块化设计便于功能扩展</li>
</ul>
<h2 id="未来优化">未来优化</h2>
<ol>
<li><strong>支持更多平台</strong>：扩展到京东、拼多多等</li>
<li><strong>增加数据分析</strong>：价格趋势分析、竞品对比</li>
<li><strong>优化用户体验</strong>：Web界面、实时进度显示</li>
<li><strong>增强稳定性</strong>：更完善的错误处理</li>
</ol>
<h2 id="结语">结语</h2>
<p>这个项目从需求分析到最终实现，经历了完整的产品开发周期。通过合理的技术选型、模块化的架构设计和完善的错误处理，最终实现了一个稳定可靠的自动化系统。</p>
<p>最大的挑战是反爬虫对抗和数据解析的复杂性，通过不断调试和优化，最终找到了有效的解决方案。</p>
<p>这个项目不仅解决了实际的业务问题，也让我在自动化测试、数据处理、系统架构等方面有了更深入的理解和实践经验。</p>
]]></content:encoded></item><item><title>macOS 终端利器 iTerm2 配置指南</title><link>https://blog.heyaohua.com/posts/2025/05/iterm2-setup-guide/</link><pubDate>Mon, 26 May 2025 15:00:00 +0800</pubDate><guid>https://blog.heyaohua.com/posts/2025/05/iterm2-setup-guide/</guid><description>iTerm2 是 macOS 上最强大的终端工具之一，本文将介绍如何安装和配置 iTerm2，让它成为你的得力助手。</description><content:encoded><![CDATA[<p>iTerm2 是 macOS 上最强大的终端工具之一，本文将介绍如何安装和配置 iTerm2，让它成为你的得力助手。</p>
<h2 id="安装步骤">安装步骤</h2>
<h3 id="1-安装-iterm2">1. 安装 iTerm2</h3>
<p>有两种安装方式：</p>
<ul>
<li>直接下载：从 <a href="https://www.iterm2.com/downloads.html">iTerm2 官网</a> 下载，解压后拖到 Applications 目录</li>
<li>使用 Homebrew：</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>brew install --cask iterm2
</span></span></code></pre></div><h3 id="2-配置主题">2. 配置主题</h3>
<ol>
<li>下载 Solarized Dark theme：http://ethanschoonover.com/solarized</li>
<li>打开 iTerm2 偏好设置：<code>Command + ,</code></li>
<li>进入 Profiles -&gt; Colors -&gt; Color Presets -&gt; Import</li>
<li>导入下载的 <code>Solarized Dark.itermcolors</code> 文件</li>
<li>选择 Solarized Dark 主题</li>
</ol>
<h3 id="3-安装-oh-my-zsh">3. 安装 Oh My Zsh</h3>
<ol>
<li>一键安装：</li>
</ol>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>sh -c &#34;$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)&#34;
</span></span></code></pre></div><ol>
<li>设置 Zsh 为默认 Shell：</li>
</ol>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>chsh -s /bin/zsh
</span></span></code></pre></div><ol>
<li>配置主题：</li>
</ol>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>vim ~/.zshrc  <span style="color:#6272a4"># 修改 ZSH_THEME=&#34;agnoster&#34;</span>
</span></span></code></pre></div><h3 id="4-配置字体">4. 配置字体</h3>
<ol>
<li>下载并安装 Meslo 字体：<a href="https://github.com/powerline/fonts/blob/master/Meslo%20Slashed/Meslo%20LG%20M%20Regular%20for%20Powerline.ttf">Meslo LG M Regular for Powerline.ttf</a></li>
<li>iTerm2 配置：</li>
<li>打开偏好设置：<code>Command + ,</code></li>
<li>Profiles -&gt; Text -&gt; Font -&gt; Change Font</li>
<li>选择 &ldquo;Meslo LG M Regular for Powerline&rdquo; 字体</li>
</ol>
<h3 id="5-增强功能配置">5. 增强功能配置</h3>
<h4 id="51-语法高亮">5.1 语法高亮</h4>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span># 安装
</span></span><span style="display:flex;"><span>brew install zsh-syntax-highlighting
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span># 配置
</span></span><span style="display:flex;"><span>echo &#34;source /usr/local/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh&#34; &gt;&gt; ~/.zshrc
</span></span></code></pre></div><h4 id="52-自动补全">5.2 自动补全</h4>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#6272a4"># 安装</span>
</span></span><span style="display:flex;"><span>git clone https://github.com/zsh-users/zsh-autosuggestions ~/.oh-my-zsh/custom/plugins/zsh-autosuggestions
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#6272a4"># 配置</span>
</span></span><span style="display:flex;"><span>vim ~/.zshrc
</span></span><span style="display:flex;"><span><span style="color:#6272a4"># 在 plugins 中添加 zsh-autosuggestions</span>
</span></span><span style="display:flex;"><span><span style="color:#8be9fd;font-style:italic">plugins</span><span style="color:#ff79c6">=(</span>git zsh-autosuggestions zsh-syntax-highlighting<span style="color:#ff79c6">)</span>
</span></span></code></pre></div><h4 id="53-快捷键配置">5.3 快捷键配置</h4>
<ol>
<li>左右键跳转：</li>
<li>打开偏好设置：<code>Command + ,</code></li>
<li></li>
</ol>
<p>Profiles → Keys → Load Preset&hellip; → Natural Text Editing</p>
<ol start="4">
<li></li>
</ol>
<p>快速显示/隐藏：</p>
<ol start="5">
<li>打开偏好设置：<code>Command + ,</code></li>
<li>Keys → Hotkey</li>
<li>设置快捷键（推荐 Command + .）</li>
</ol>
<h4 id="54-隐藏用户名和主机名">5.4 隐藏用户名和主机名</h4>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span># 在 ~/.zshrc 中添加
</span></span><span style="display:flex;"><span>DEFAULT_USER=&#34;heyaohua&#34;
</span></span></code></pre></div><h4 id="55-配置代理">5.5 配置代理</h4>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span># 在 ~/.zshrc 中添加
</span></span><span style="display:flex;"><span>alias proxy=&#39;export all_proxy=socks5://127.0.0.1:1086&#39;
</span></span><span style="display:flex;"><span>alias unproxy=&#39;unset all_proxy&#39;
</span></span></code></pre></div><h2 id="使用技巧">使用技巧</h2>
<ol>
<li>分屏操作：</li>
<li>垂直分屏：Command + D</li>
<li>水平分屏：Command + Shift + D</li>
<li></li>
</ol>
<p>切换分屏：Command + [ 或 Command + ]</p>
<ol start="5">
<li></li>
</ol>
<p>标签页操作：</p>
<ol start="6">
<li>新建标签页：Command + T</li>
<li>关闭标签页：Command + W</li>
<li></li>
</ol>
<p>切换标签页：Command + 数字 或 Command + 左右方向键</p>
<ol start="9">
<li></li>
</ol>
<p>其他常用快捷键：</p>
<ol start="10">
<li>清屏：Command + K</li>
<li>查找：Command + F</li>
<li>复制：Command + C</li>
<li>粘贴：Command + V</li>
<li>全屏：Command + Enter</li>
</ol>
<h2 id="常见问题解决">常见问题解决</h2>
<ol>
<li>如果自动补全颜色不明显：</li>
<li>打开偏好设置：<code>Command + ,</code></li>
<li>Profiles -&gt; Colors</li>
<li></li>
</ol>
<p>修改 ANSI Colors 中的颜色值（推荐：586e75）</p>
<ol start="5">
<li></li>
</ol>
<p>如果出现乱码：</p>
<ol start="6">
<li>确保已安装 Powerline 字体</li>
<li>检查 iTerm2 的字体设置是否正确</li>
<li>重新加载 zsh 配置：<code>source ~/.zshrc</code></li>
</ol>
]]></content:encoded></item><item><title>抖音长截图服务使用说明</title><link>https://blog.heyaohua.com/posts/2024/01/douyin-screenshot-service/</link><pubDate>Mon, 15 Jan 2024 16:30:00 +0800</pubDate><guid>https://blog.heyaohua.com/posts/2024/01/douyin-screenshot-service/</guid><description>本文介绍基于 Playwright 封装的抖音长截图服务，涵盖能力范围、接口设计、使用范例以及性能调优建议，便于快速集成到现有业务系统中。</description><content:encoded><![CDATA[<p>本文介绍基于 Playwright 封装的抖音长截图服务，涵盖能力范围、接口设计、使用范例以及性能调优建议，便于快速集成到现有业务系统中。</p>
<h2 id="服务概述">服务概述</h2>
<p>核心功能已经封装在 <code>app/services/playwright_service.py</code> 中，经过大量测试验证，推荐使用的默认参数如下：</p>
<ul>
<li>滚动距离：1000 px</li>
<li>底部裁剪：300 px</li>
<li>平均截图数量：4–5 张</li>
<li>平均文件大小：2–4 MB</li>
</ul>
<blockquote>
<p>GitHub 仓库：<a href="https://github.com/heyaohua/douyin_screenshot_service">https://github.com/heyaohua/douyin_screenshot_service</a></p>
</blockquote>
<h2 id="核心能力">核心能力</h2>
<ul>
<li><strong>移动端模拟</strong>：完整模拟 iPhone 设备视口与 UA</li>
<li><strong>智能滚动</strong>：自动识别内部滚动容器并控制节奏</li>
<li><strong>懒加载处理</strong>：触发页面动态内容渲染</li>
<li><strong>图像拼接</strong>：裁剪重叠区域，生成无缝长图</li>
<li><strong>错误处理</strong>：封装异常重试、日志与超时机制</li>
</ul>
<h3 id="主要接口">主要接口</h3>
<h4 id="take_long_screenshoturl-output_dirscreenshots"><code>take_long_screenshot(url, output_dir=&quot;screenshots&quot;)</code></h4>
<ul>
<li><code>url</code>：待截图抖音页面 URL</li>
<li><code>output_dir</code>：输出目录（默认 <code>screenshots</code>）</li>
</ul>
<p>返回值示例：</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;success&#34;</span>: True,
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;output_path&#34;</span>: <span style="color:#f1fa8c">&#34;screenshots/douyin_long_screenshot_20250916_135333.png&#34;</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;screenshot_count&#34;</span>: <span style="color:#bd93f9">4</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;total_height&#34;</span>: <span style="color:#bd93f9">9228</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;file_size&#34;</span>: <span style="color:#bd93f9">2764800</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;original_url&#34;</span>: <span style="color:#f1fa8c">&#34;https://v.douyin.com/...&#34;</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;current_url&#34;</span>: <span style="color:#f1fa8c">&#34;https://haohuo.jinritemai.com/...&#34;</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;title&#34;</span>: <span style="color:#f1fa8c">&#34;页面标题&#34;</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><h2 id="http-api-设计">HTTP API 设计</h2>
<h3 id="长截图接口">长截图接口</h3>
<ul>
<li><strong>请求</strong>：<code>POST /douyin/long-screenshot</code></li>
<li><strong>请求体</strong>：</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>  <span style="color:#ff79c6">&#34;url&#34;</span>: <span style="color:#f1fa8c">&#34;https://v.douyin.com/Zdo3P7Zv51o/&#34;</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><ul>
<li><strong>响应</strong>：</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>  <span style="color:#ff79c6">&#34;message&#34;</span>: <span style="color:#f1fa8c">&#34;长截图完成&#34;</span>,
</span></span><span style="display:flex;"><span>  <span style="color:#ff79c6">&#34;data&#34;</span>: {
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;success&#34;</span>: <span style="color:#ff79c6">true</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;output_path&#34;</span>: <span style="color:#f1fa8c">&#34;screenshots/douyin_long_screenshot_20250916_135333.png&#34;</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;screenshot_count&#34;</span>: <span style="color:#bd93f9">4</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;total_height&#34;</span>: <span style="color:#bd93f9">9228</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">&#34;file_size&#34;</span>: <span style="color:#bd93f9">2764800</span>
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><h3 id="测试接口">测试接口</h3>
<ul>
<li><strong>请求</strong>：<code>POST /douyin/test-long-screenshot</code></li>
<li><strong>说明</strong>：使用预设 URL 触发一次完整流程，便于巡检。</li>
</ul>
<h2 id="使用示例">使用示例</h2>
<h3 id="服务内调用">服务内调用</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#ff79c6">from</span> app.services.playwright_service <span style="color:#ff79c6">import</span> playwright_service
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff79c6">async</span> <span style="color:#ff79c6">def</span> <span style="color:#50fa7b">example</span>():
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">await</span> playwright_service<span style="color:#ff79c6">.</span>initialize()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    result <span style="color:#ff79c6">=</span> <span style="color:#ff79c6">await</span> playwright_service<span style="color:#ff79c6">.</span>take_long_screenshot(
</span></span><span style="display:flex;"><span>        url<span style="color:#ff79c6">=</span><span style="color:#f1fa8c">&#34;https://v.douyin.com/Zdo3P7Zv51o/&#34;</span>,
</span></span><span style="display:flex;"><span>        output_dir<span style="color:#ff79c6">=</span><span style="color:#f1fa8c">&#34;screenshots&#34;</span>
</span></span><span style="display:flex;"><span>    )
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">if</span> result[<span style="color:#f1fa8c">&#34;success&#34;</span>]:
</span></span><span style="display:flex;"><span>        <span style="color:#8be9fd;font-style:italic">print</span>(<span style="color:#f1fa8c">f</span><span style="color:#f1fa8c">&#34;截图完成: </span><span style="color:#f1fa8c">{</span>result[<span style="color:#f1fa8c">&#39;output_path&#39;</span>]<span style="color:#f1fa8c">}</span><span style="color:#f1fa8c">&#34;</span>)
</span></span><span style="display:flex;"><span>        <span style="color:#8be9fd;font-style:italic">print</span>(<span style="color:#f1fa8c">f</span><span style="color:#f1fa8c">&#34;截图数量: </span><span style="color:#f1fa8c">{</span>result[<span style="color:#f1fa8c">&#39;screenshot_count&#39;</span>]<span style="color:#f1fa8c">}</span><span style="color:#f1fa8c">&#34;</span>)
</span></span><span style="display:flex;"><span>        <span style="color:#8be9fd;font-style:italic">print</span>(<span style="color:#f1fa8c">f</span><span style="color:#f1fa8c">&#34;总高度: </span><span style="color:#f1fa8c">{</span>result[<span style="color:#f1fa8c">&#39;total_height&#39;</span>]<span style="color:#f1fa8c">}</span><span style="color:#f1fa8c">px&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#ff79c6">await</span> playwright_service<span style="color:#ff79c6">.</span>close()
</span></span></code></pre></div><h3 id="http-api-调用">HTTP API 调用</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -X POST <span style="color:#f1fa8c">&#34;http://localhost:8000/douyin/long-screenshot&#34;</span> <span style="color:#f1fa8c">\
</span></span></span><span style="display:flex;"><span>     -H <span style="color:#f1fa8c">&#34;Content-Type: application/json&#34;</span> <span style="color:#f1fa8c">\
</span></span></span><span style="display:flex;"><span>     -d <span style="color:#f1fa8c">&#39;{&#34;url&#34;: &#34;https://v.douyin.com/Zdo3P7Zv51o/&#34;}&#39;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>curl -X POST <span style="color:#f1fa8c">&#34;http://localhost:8000/douyin/test-long-screenshot&#34;</span>
</span></span></code></pre></div><h3 id="本地测试脚本">本地测试脚本</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#8be9fd;font-style:italic">cd</span> tests
</span></span><span style="display:flex;"><span>python test_service.py
</span></span></code></pre></div><h2 id="性能指标">性能指标</h2>
<table>
  <thead>
      <tr>
          <th>滚动距离</th>
          <th>截图数量</th>
          <th>平均文件大小</th>
          <th>效率评估</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>1000 px</td>
          <td>4–5 张</td>
          <td>2.6–3.7 MB</td>
          <td>⭐⭐⭐⭐⭐ 推荐参数</td>
      </tr>
      <tr>
          <td>900 px</td>
          <td>6 张</td>
          <td>4.6 MB</td>
          <td>⭐⭐⭐⭐</td>
      </tr>
      <tr>
          <td>800 px</td>
          <td>6 张</td>
          <td>4.3 MB</td>
          <td>⭐⭐⭐</td>
      </tr>
      <tr>
          <td>700 px</td>
          <td>7 张</td>
          <td>5.0 MB</td>
          <td>⭐⭐</td>
      </tr>
      <tr>
          <td>600 px</td>
          <td>8 张</td>
          <td>5.7 MB</td>
          <td>⭐</td>
      </tr>
  </tbody>
</table>
<h2 id="目录结构">目录结构</h2>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>app/services/
</span></span><span style="display:flex;"><span>├── playwright_service.py     # 主服务入口
</span></span><span style="display:flex;"><span>│   ├── take_long_screenshot  # 长截图方法
</span></span><span style="display:flex;"><span>│   └── _stitch_screenshots   # 图片拼接方法
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>app/api/
</span></span><span style="display:flex;"><span>├── douyin.py                 # HTTP API 定义
</span></span><span style="display:flex;"><span>│   ├── /long-screenshot      # 长截图接口
</span></span><span style="display:flex;"><span>│   └── /test-long-screenshot # 测试接口
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>tests/
</span></span><span style="display:flex;"><span>├── test_service.py           # 集成测试脚本
</span></span><span style="display:flex;"><span>└── simple_douyin_test.py     # 快速验证脚本
</span></span></code></pre></div><h2 id="运维与实践建议">运维与实践建议</h2>
<ol>
<li><strong>生命周期管理</strong>：调用前需执行 <code>initialize()</code>，结束后务必调用 <code>close()</code> 释放浏览器资源。</li>
<li><strong>目录权限</strong>：确保输出目录具备写权限，并定期清理历史截图。</li>
<li><strong>错误兜底</strong>：检查返回的 <code>success</code> 字段并结合日志定位问题。</li>
<li><strong>并发控制</strong>：视负载设置队列或限流，避免同时创建大量浏览器实例。</li>
<li><strong>监控告警</strong>：建议将成功率、耗时、文件大小等指标纳入监控系统。</li>
</ol>
<h2 id="日志样例">日志样例</h2>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>2025-09-16 13:53:33,832 - INFO - 拼接图片尺寸: 1170 x 9228
</span></span><span style="display:flex;"><span>2025-09-16 13:53:35,160 - INFO - ✅ 成功: True
</span></span><span style="display:flex;"><span>2025-09-16 13:53:35,160 - INFO - 📊 截图数量: 4
</span></span><span style="display:flex;"><span>2025-09-16 13:53:35,160 - INFO - 💾 文件大小: 2.64MB
</span></span></code></pre></div><h2 id="成功案例">成功案例</h2>
<ul>
<li>URL：<code>https://v.douyin.com/Zdo3P7Zv51o/</code></li>
<li>截图数量：4 张</li>
<li>总高度：9228 px</li>
<li>文件大小：2.64 MB</li>
<li>处理时长：约 22 秒</li>
<li>成功率：100%</li>
</ul>
<hr>
<p>如需深入定制或扩展能力，请参考仓库中的完整代码与测试用例。</p>
]]></content:encoded></item><item><title>Python开发小技巧分享</title><link>https://blog.heyaohua.com/posts/2024/01/python-tips/</link><pubDate>Mon, 15 Jan 2024 14:30:00 +0800</pubDate><guid>https://blog.heyaohua.com/posts/2024/01/python-tips/</guid><description>日常开发中，掌握一些高频技巧能够明显提升代码质量与效率。本文整理了五个常用的小窍门，并配以示例代码，便于在项目中直接应用。</description><content:encoded><![CDATA[<p>日常开发中，掌握一些高频技巧能够明显提升代码质量与效率。本文整理了五个常用的小窍门，并配以示例代码，便于在项目中直接应用。</p>
<h2 id="1-善用列表推导式">1. 善用列表推导式</h2>
<p>列表推导式可以将循环与条件判断浓缩到一行，既简洁又易读：</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span># 传统写法
</span></span><span style="display:flex;"><span>squares = []
</span></span><span style="display:flex;"><span>for x in range(10):
</span></span><span style="display:flex;"><span>    squares.append(x**2)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span># 列表推导式
</span></span><span style="display:flex;"><span>squares = [x**2 for x in range(10)]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span># 搭配条件过滤
</span></span><span style="display:flex;"><span>even_squares = [x**2 for x in range(10) if x % 2 == 0]
</span></span></code></pre></div><h2 id="2-使用-dictget-提升容错性">2. 使用 <code>dict.get</code> 提升容错性</h2>
<p>通过 <code>dict.get</code> 读取字典时，可定义默认值，避免 KeyError 并简化分支逻辑：</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span># 可能抛出 KeyError
</span></span><span style="display:flex;"><span>user_name = user_dict[&#39;name&#39;]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span># 更稳健的写法
</span></span><span style="display:flex;"><span>user_name = user_dict.get(&#39;name&#39;, &#39;Unknown&#39;)
</span></span></code></pre></div><h2 id="3-搭配-enumerate-获取索引">3. 搭配 <code>enumerate</code> 获取索引</h2>
<p><code>enumerate</code> 能在遍历序列时同时获得索引和值，避免手动维护计数器：</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span>fruits <span style="color:#ff79c6">=</span> [<span style="color:#f1fa8c">&#39;apple&#39;</span>, <span style="color:#f1fa8c">&#39;banana&#39;</span>, <span style="color:#f1fa8c">&#39;orange&#39;</span>]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff79c6">for</span> index, fruit <span style="color:#ff79c6">in</span> <span style="color:#8be9fd;font-style:italic">enumerate</span>(fruits, start<span style="color:#ff79c6">=</span><span style="color:#bd93f9">1</span>):
</span></span><span style="display:flex;"><span>    <span style="color:#8be9fd;font-style:italic">print</span>(<span style="color:#f1fa8c">f</span><span style="color:#f1fa8c">&#34;</span><span style="color:#f1fa8c">{</span>index<span style="color:#f1fa8c">}</span><span style="color:#f1fa8c">. </span><span style="color:#f1fa8c">{</span>fruit<span style="color:#f1fa8c">}</span><span style="color:#f1fa8c">&#34;</span>)
</span></span></code></pre></div><h2 id="4-用-zip-打包多组数据">4. 用 <code>zip</code> 打包多组数据</h2>
<p>当需要并行遍历多个可迭代对象时，<code>zip</code> 能有效避免索引操作：</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span>names <span style="color:#ff79c6">=</span> [<span style="color:#f1fa8c">&#39;Alice&#39;</span>, <span style="color:#f1fa8c">&#39;Bob&#39;</span>, <span style="color:#f1fa8c">&#39;Charlie&#39;</span>]
</span></span><span style="display:flex;"><span>ages <span style="color:#ff79c6">=</span> [<span style="color:#bd93f9">25</span>, <span style="color:#bd93f9">30</span>, <span style="color:#bd93f9">35</span>]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff79c6">for</span> name, age <span style="color:#ff79c6">in</span> <span style="color:#8be9fd;font-style:italic">zip</span>(names, ages):
</span></span><span style="display:flex;"><span>    <span style="color:#8be9fd;font-style:italic">print</span>(<span style="color:#f1fa8c">f</span><span style="color:#f1fa8c">&#34;</span><span style="color:#f1fa8c">{</span>name<span style="color:#f1fa8c">}</span><span style="color:#f1fa8c"> 的年龄是 </span><span style="color:#f1fa8c">{</span>age<span style="color:#f1fa8c">}</span><span style="color:#f1fa8c"> 岁&#34;</span>)
</span></span></code></pre></div><h2 id="5-借助-f-string-优雅格式化">5. 借助 f-string 优雅格式化</h2>
<p>Python 3.6 及以上版本推荐使用 f-string 进行字符串拼接，可读性高、性能更优：</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>name = &#34;World&#34;
</span></span><span style="display:flex;"><span>age = 25
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>message = f&#34;Hello, {name}! You are {age} years old.&#34;
</span></span></code></pre></div><hr>
<p>这些技巧虽然简单，却能在编写脚本、数据处理或后端服务时显著提升编码体验。欢迎在评论中分享你常用的 Python 诀窍。</p>
]]></content:encoded></item></channel></rss>